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Executive  Summary 


The  Navy  trains  thousands  of  sailors  each  year  in  the  individual  skills  required  to 
do  their  jobs.  The  cost  of  training  these  students  in  institutional  settings  is  enor¬ 
mous;  it  consists  of  classroom  direct  and  overhead  costs,  student  and  instructor 
pay  and  allowances,  and  courseware  acquisition  and  maintenance,  among  other 
costs.  The  application  of  new  training  technologies  promises  to  reduce  some  of 
these  costs.  The  Navy,  like  its  Service  counterparts,  is  searching  for  the  technol¬ 
ogy  or  combination  of  technologies  that  will  provide  the  greatest  training  benefit 
and  reduce  the  costs  of  training  sailors. 

Interactive  electronic  technical  manuals  (IETMs)  are  devices  that  originally  were 
developed  to  aid  technicians  in  the  performance  of  their  jobs  by  accelerating  in¬ 
formation  retrieval,  aiding  equipment  diagnosis  and  repair,  and  facilitating  re¬ 
cordkeeping  and  logistics  management.  Navy  trainers  have  discovered  that 
IETMs  also  can  help  the  technician  learn.  The  Navy  wants  to  exploit  this  discov¬ 
ery  by  characterizing  and  developing  the  training  potential  of  IETMs  with  the  aim 
of  more  efficiently  training  technicians  and,  ultimately,  saving  Navy  time  and  re¬ 
sources. 

The  Navy  asked  the  Logistics  Management  Institute  to  characterize  how  the  Navy 
acquires  IETMs,  how  they  are  used,  and  what  resource  impacts  result  from  their 
acquisition  and  use  in  training  venues.  We  were  asked  to  assess  the  potential  costs 
and  benefits  to  the  Navy  of  acquiring  these  devices,  not  only  from  an  operational 
perspective  but  from  the  training  and  education  perspective  as  well.  We  were  also 
asked  to  build  a  computer-based  tool  to  help  the  N81  analyst  assess  the  relative 
cost-benefits  of  alternative  IETM  programs. 

Our  research  found  that,  although  there  is  general  guidance  asking  that  program 
offices  acquire  digital  technical  information  where  it  is  deemed  “cost  effective,” 
there  is  no  accompanying  guidance  on  how  to  measure  cost  effectiveness.  Fur¬ 
thermore,  there  is  almost  no  binding  direction  regarding  the  form  or  function  of 
IETMs.  Hence,  program  offices  are  acting  autonomously — and  the  result  is  that 
the  Navy  is  buying  devices  with  many  widely  differing  formats  and  capabilities. 
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Also,  logistics  acquisition  planners  and  training  planners  work  in  distinct  stove¬ 
pipes”  that  hinder  collaboration  on  IETM  development. 

We  found  that  the  predominant  incentive  for  buying  IETMs  appears  to  be  their 
potential  savings  in  production,  distribution,  and  maintenance  relative  to  paper 
manuals.  Because  the  cost  of  technical  manual  development  and  conversion  in¬ 
creases  tremendously  with  the  functionality  of  the  device,  most  acquisition  man¬ 
agers  are  buying  the  lower  end,  or  Class  1  and  2,  IETMs. 

There  are  several  suppositions  about  the  impact  that  IETMs  could  have  on  insti¬ 
tutional  training  and  job-site  training  and  operations.  Unfortunately,  there  is  very 
little  empirical  data  with  which  to  substantiate  the  claims.  We  gathered  anecdotal 
evidence,  the  assessment  of  which  leads  us  to  conclude  that  there  are  indeed  sav¬ 
ings  and  other  benefits  to  be  gained  by  using  IETMs  as  training  tools.  Many  re¬ 
sults  were  mixed,  however,  and  the  magnitude  of  the  IETM’s  impact  often  was 
overstated.  To  date,  there  has  been  little  documented  improvement  in  “street-to- 
fleef  ’  time — a  prime  indicator  of  training  efficiency — for  apprentice-level  techni¬ 
cians. 

Based  on  these  findings,  we  conclude  that  a  focused  pilot  study  is  needed  to  col¬ 
lect  data  with  which  to  quantify  the  impact  of  IETMs  and  other  training  technolo¬ 
gies.  We  also  conclude  that  an  assessment  of  emerging  training  technologies  and 
competing  plans  and  guidance  is  essential.  The  assessment  would  help  to  create  a 
Navy  enterprise  plan  for  training  technology  development  and  deployment. 
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Chapter  1 

Introduction 

Background 

Technical  manuals  (TMs)  are  publications  that  contain  instructions  for  the  instal¬ 
lation,  maintenance,  and  support  of  defense  systems,  system  components,  and 
support  equipment.  TMs  can  be  traditional  paper  documents;  electronic  technical 
manuals  (ETMs),  which  display  the  traditional  page-based  information  on  a  com¬ 
puter  screen;  or  interactive  electronic  technical  manuals  (IETMs),  which  are  smart 
databases  of  text  and  graphics,  arranged  and  presented  in  pageless,  frame- 
oriented,  interactive  formats.1 

IETMs  streamline  access  to  information  and  provide  multiple  paths  to  related  in¬ 
formation.  Depending  on  the  level  of  sophistication,  an  IETM  can  duplicate  on  a 
personal  computer  or  other  display  device  the  research  environment  available  in  a 
well-equipped  reference  library.  Powerful  interactive  troubleshooting  procedures 
and  tutorials,  which  are  not  possible  with  paper  technical  manuals,  can  be  made 
available  by  using  the  intelligent  features  of  the  IETM  display  device. 

The  entire  Department  of  Defense  (DoD)  is  moving  to  digitize  as  much  technical 
information  and  logistical  data  as  possible.  This  effort  includes  a  mandate  to  pro¬ 
cure  or  convert  equipment  TMs  into  digital  formats  wherever  practical.  The  TM 
digitization  movement  initially  was  driven  by  the  fact  that  printing,  distributing, 
maintaining,  and  storing  paper  technical  publications  was  consuming  increasing 
resources  and  using  large  amounts  of  infrastructure.  The  digitization  effort  also 
has  been  spurred  by  research  that  indicates  that  digital  information  systems  are 
not  only  less  costly  but  more  effective  and  efficient  to  use  in  their  primary  role  of 
supporting  system  operations  and  maintenance  than  paper-based  products." 

There  also  is  an  emerging  body  of  research  that  suggests  that  prudent  and  effec¬ 
tive  employment  of  interactive  courseware  (ICW)  in  classrooms  can  enable 
courses  to  reduce  in-classroom  time  by  10-40  percent.  Moreover,  when  trainees 
learn  at  their  own  pace,  they  learn  better,  retain  more,  and  understand  when  and 
how  to  apply  what  they  have  learned. 

The  Navy  wants  to  leverage  all  of  these  potential  benefits  of  IETMs  and  ICW  to 
achieve  maximum  effectiveness  and  efficiency  from  its  training  and  its  opera- 


1  Office  of  the  Secretary  of  Defense,  Program  Manager’s  Desktop  Guide  for  Continuous  Ac¬ 
quisition  and  Life-Cycle  Support  (CALS)  Implementation,  1997. 

2  Eric  L.  Jorgensen,  The  Move  to  Paperless  Technical  Manuals  in  the  US  DoD,  Naval  Surface 
Warfare  Center,  Carderock  Division,  undated. 
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tions.  The  Assessment  Division  (N81)  within  the  Office  of  the  Chief  of  Naval 
Operations  (OPNAV)  asked  LMI  to  assess  the  potential  costs  and  benefits  of  ac¬ 
quiring  IETMs  for  new  and  legacy  systems.  Unlike  previous  research  in  this  field, 
however,  this  investigation  assesses  this  potential  from  the  training  and  education 
perspective.  N81  theorizes  that  the  prudent  application  of  EETMs  will  reap  mone¬ 
tary  savings  and  operational  benefits  for  the  training  and  education  environment 
as  well  as  for  the  fleet. 

Purpose 


The  purpose  of  this  research  is  to 

♦  assess  the  potential  costs  and  benefits  to  the  Navy  of  acquiring  IETMs, 

♦  characterize  how  the  Navy  acquires  IETMs, 

♦  characterize  how  the  Navy  uses  IETMs, 

♦  develop  an  analytical  tool  for  analyzing  alternative  IETM  programs,  and 

♦  describe  the  resource  impacts  that  result  from  IETM  acquisition  and  use. 

Discussion  of  the  analytical  tool  can  be  found  in  LMI  Report  NA008T2,  Interac¬ 
tive  Electronic  Technical  Manual  Cost-Benefit  Analysis  Tool  User’s  Guide. 

Approach 

First  we  conducted  an  extensive  literature  and  media  search  and  interviewed  poli¬ 
cymakers  in  the  Navy  to  understand  what  the  Service  is  doing  with  regard  to  ac¬ 
quiring  IETMs — who  manages  the  process,  what  rules  and  standards  govern  the 
process,  and  what  goals  or  imperatives  drive  the  Navy’s  policies. 

Next  we  gathered  and  assessed  quantitative  and  qualitative  information  about  the 
use  of  these  devices  in  institutional  training  settings  and  at  the  Fleet.  The  magni¬ 
tude  of  the  data  deficit  became  clear  during  this  phase  of  the  study.  There  is  no 
central  repository  of  IETMs  experiential  data  (or  even  repositories  local  to  sys¬ 
tems  commands),  so  we  captured  anecdotal  data  by  studying  a  few  high-visibility 
cases.  We  refer  to  the  experiences  of  these  cases  frequently  in  this  report. 

We  also  assessed  some  selected  sample  data  from  the  incorporation  of  IETMs  into 
Advanced  Electronic  Classroom  (AEC)  technology  suites.  Those  data  give  us  a 
look  at  the  potential  impacts  of  IETMs  in  institutional  training. 

Finally,  we  developed  an  analytical  framework  to  help  analysts  compare  alterna¬ 
tive  technology  acquisition  programs.  Although  the  model  was  developed  with 
IETMs  in  mind,  the  basic  framework  can  be  re-used  to  conduct  many  analyses, 
such  as  alternative  classroom  technology  or  distance  learning  assessments.  The 
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model  and  its  documentation  are  being  provided  to  N81  under  separate  cover  (see 
LMI  Report  NA008T2). 

Challenges  and  Caveats 

♦  According  to  staff  in  the  Office  of  the  Director  of  Naval  Training  (N75), 
“To  date,  there  are  no  good  studies  that  show  costs-benefits  of  the  appli¬ 
cation  of  IETMs  to  training.”3  Neither  of  the  military  services  has  con¬ 
tracted  for  in-depth  study  of  the  potential  of  IETMs — in  part  because  it 
would  take  significant  time  and  money  to  study  the  issues.  Moreover,  no 
two  programs  appear  to  use  the  same  analytical  justification  for  acquiring 
IETMs.  And,  as  we  found  during  our  research,  there  has  been  little  con¬ 
certed  effort  to  collect  data  on  IETM  procurement  or  implementation. 

♦  Empirical  data  on  the  use  of  IETMs  in  the  Navy  are  nearly  nonexistent. 
Experience  data  for  IETMs  use,  either  in  schoolhouses  or  operational  set¬ 
tings,  have  not  been  collected  to  any  significant  degree.  We  assess  anec¬ 
dotally  the  costs  and  benefits  for  the  Navy,  from  training  and  education  as 
well  as  organizational-level  logistics  perspectives,  of  acquiring  IETMs  in 
lieu  of  paper  TMs. 

♦  We  acknowledge  that  current  DoD  and  Navy  directives  call  for  all  acqui¬ 
sition  programs  to  procure  or  convert  all  technical  information  into  digital 
formats  for  delivery.4  Because  all  of  DoD  is  moving  toward  the  use  of 
digital  technical  information  for  all  new  and  legacy  equipment,  maintain¬ 
ing  paper  manuals  is  an  unrealistic  alternative;  we  used  that  scenario  in 
this  assessment  only  as  a  statement  of  the  status  quo.  Because  program  of¬ 
fices  are  required  to  assess  the  cost-effectiveness  of  acquiring  TMs  in 
digital  form,  however,  our  assessment  compares  IETM  classes  to  paper- 
based  operations  and  training. 

♦  We  do  not  address  requirements  determination  in  our  discussion.  We  start 
our  analysis  from  the  assumption  that  the  argument  already  has  been  made 
to  procure  IETMs.  We  understand  that  ideally,  IETM  class  selection  and 
functionality  would  be  driven  by  the  operational  requirements  spelled  out 
in  the  basic  system’s  concept  of  operations  (CONOPS). 

♦  We  do  not  address  the  cost  of  training  infrastructure  (i.e.,  schoolhouse  fa¬ 
cilities)  in  our  assessment.  Costs  associated  with  institutional  training  fa¬ 
cilities  are  assumed  to  be  sunk  costs. 


3  Interview  with  Tim  Tate,  N75, 2  March  2000. 

4  DoD  Regulation  5000.2-R  and  DON's  Guidance  on  Acquisition  and  Conversion  of  Logistics 
Technical  Data  to  Digital  Form,  July  1999,  dictate  that  technical  data  for  new  systems  be  pro¬ 
cured  in  digital  form  and  that  data  for  legacy  systems  be  converted  to  digital  form  “when  conver¬ 
sion  is  cost-effective.” 
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♦  Whether  procuring  a  new  IETM  or  considering  conversion  of  an  existing 
TM  to  an  interactive  digital  format,  the  development  and  procurement 
costs  are  only  a  fraction  of  the  total  life-cycle  cost  of  the  TM.  There  are 
costs  associated  with  incorporating  the  devices  into  classroom  technology 
suites,  costs  to  maintain  and  update  the  device  and  its  data,  and  costs  and 
benefits  of  using  the  devices  to  train  students  and  maintain  equipment.  No 
existing  cost  model  facilitates  a  rigorous  analysis  to  select  the  optimal, 
most  cost-effective  level  of  TM  automation  that  address  each  of  these 
components.  Unfortunately,  this  situation  tends  to  focus  attention  on  the_ 
costs  of  conversion  rather  than  a  true  understanding  of  the  total  impacts. 


IETM  Technology 

As  a  benefit  to  the  reader,  we  provide  a  basic  understanding  of  IETM  technology 
before  proceeding  to  the  remainder  of  this  report.  The  following  subsections  de¬ 
scribe  the  basic  features  and  capabilities  of  these  devices. 


General  Description 

An  IETM  is  a  computer-based  collection  of  information  needed  for  the  diagnosis 
and  maintenance  of  a  defense  system.  An  IETM  enables  the  user  to  locate  re¬ 
quired  information  faster  and  more  easily  than  is  possible  with  a  paper  technical 
manual.  When  IETMs  are  designed  to  DoD  specifications,  they  are  easier  to  com¬ 
prehend,  more  specifically  matched  to  the  system  configuration  under  diagnosis, 
and  available  in  a  form  that  requires  much  less  physical  storage  space  than  paper. 

In  general,  an  IETM  is  optically  arranged  and  formatted  for  interactive  presenta¬ 
tion  to  the  end  user  on  an  electronic  display  system.  Unlike  other  optical  systems 
that  display  a  page  of  text  from  a  single  document,  IETMs  can  present  interrelated 
information  from  multiple  sources  tailored  to  user  queries. 

IETM  Classes 

Although  IETMs  generally  are  easier  to  use  than  paper  manuals,  there  is  a  wide 
range  of  capability  and  usability.  These  capabilities  are  captured  in  categories 
called  classes.  Table  1-1  provides  a  description  of  the  classes. 


5  Commander,  Naval  Sea  Systems  Command  and  Commander,  Space  and  Naval  Warfare 
Systems  Command,  Interactive  Electronic  Technical  Manual  (IETM)  Process  Plan,  5  December 
1995. 
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Introduction 


Table  1-1.  IETM  Classes 


IETM 

class 

Description 

0 

Non-electronically  indexed  page  images 

1 

Electronically  indexed  pages  images 

2 

Electronic  scrolling  documents 

3 

Linear  structured  IETMs 

4 

Hierarchically  structured  IETMs 

5 

Integrated  database  IETIS 

The  lowest  classes  of  IETMs  usually  are  referred  to  as  ETMs  instead  of  IETMs 
because  they  offer  no  interactivity;  they  are  merely  digitized  images  of  paper 
pages.  Class  0  has  no  indexing  whatsoever,  so  the  user  must  view  page  after  page 
to  access  any  information.  Class  1  is  similar  to  Class  0  but  may  include  a  basic 
level  of  indexing  that  enables  the  user  to  “jump”  to  desired  chapters  or  sections. 
Classes  2  and  3  incorporate  the  basic  functionality  we  typically  require  of  an 
IETM.  Class  2  IETMs  usually  consist  of  American  Standard  Code  for  Information 
Interchange  (ASCII)  or  portable  data  format  (PDF)  files  in  a  linear  format  that 
offers  the  user  the  capability  to  search,  browse,  and  hyperlink  to  selected  topics. 
Class  3  adds  the  first  real  interactivity  by  including  dialog  boxes.  Class  3  data 
usually  are  in  Standard  Generalized  Markup  Language  (SGML)  format,  with 
content  tags  rather  than  the  format  tags  found  in  the  Class  2  device.  Class  3  also 
adds  the  logical  “next”  and  “back”  functions  to  the  device’s  capability. 

The  real  advance  in  IETM  formats  occurs  between  Class  3  and  Class  4.  Class  4 
and  5  devices  use  database  structures  rather  than  the  linear  document  structures  of 
the  lower  classes.  This  arrangement  allows  for  much  more  flexible  cross- 
referencing.  The  interactive  features  of  the  Class  4  device  usually  are  integral  to 
the  authoring  process  rather  than  added  on  after  the  content  is  written.  Finally,  the 
Class  5  device  really  is  a  multifunctional  integrated  electronic  technical  informa¬ 
tion  system  (IETIS)  that  enables  the  user  to  interface  with  other  databases,  main¬ 
tenance  and  diagnostic  systems,  administrative  and  logistical  systems,  and  more. 

The  higher  the  class,  the  more  technologically  complex  the  IETM.  Early  work  by 
researchers  at  the  Naval  Surface  Warfare  Center-Carderock  Division  (NSWC- 
CD)  demonstrated  the  benefits  of  IETMs  over  paper-based  manuals.  The  benefits 
increase  with  each  higher  class.  The  lowest  classes  (0  and  1)  have  correspond¬ 
ingly  lower  production  costs  because  for  the  most  part  they  are  merely  electronic 
versions  of  the  paper  products.  The  higher  classes  (4  and  5)  have  the  greatest  po¬ 
tential  to  affect  how  maintenance  training  takes  place.  We  address  this  potential  in 
greater  detail  in  subsequent  chapters  of  this  report. 


6  Draft  Navy  Interactive  Electronic  Technical  Manual  (IETM)  Process  Plan  (IETMPP),  54. 
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Organization  of  the  Report 

The  remainder  of  this  report  is  structured  as  follows: 

♦  Chapter  2  reports  the  findings  of  our  background  research  into  IETMs.  We 
describe  the  Navy’s  organizations  and  processes  that  affect  how  IETMs 
are  developed  and  acquired. 

♦  Chapter  3  presents  the  IETM  acquisition  and  cost  experiences  of  some 
high-profile  programs.  It  also  describes  the  processes  and  costs  involved 
in  converting  from  paper  manuals  to  ETMs  or  IETMs. 

♦  Chapter  4  focuses  on  Navy  training  and  the  potential  that  IETMs  offer  to 
make  training  more  cost  effective. 

♦  Chapter  5  recaps  our  analysis  and  findings  and  discusses  the  insights  gen¬ 
erated  by  the  findings.  We  draw  conclusions  about  what  actions  the  Navy 
might  wish  to  take  to  understand  and  realize  the  full  potential  of  IETMs. 
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Chapter  2 

IETMs  and  Navy  Acquisition 


One  of  our  primary  tasks  was  to  characterize  the  process  of  acquiring  IETMs  in 
the  Navy.  We  wanted  to  understand  who  does  what  with  regard  to  IETM  devel¬ 
opment  and  acquisition  policy  and  what  imperatives,  if  any,  Navy  acquisition 
program  offices  are  responding  to. 

In  this  chapter,  we  identify  and  evaluate  the  organizations  and  offices  responsible 
for  IETMs  development  and  acquisition.  We  also  address  some  of  the  key  issues 
Navy  programs  face  in  the  development  and  acquisition  of  IETMs. 

IETMs  Development  and  Acquisition  Policies 

Figure  2-1  shows,  very  simply,  how  the  Navy  guides  the  development  of  IETMs. 
The  process  is  not  directive;  in  fact,  there  is  very  little  binding  guidance  involved. 
Virtually  all  of  the  guidance  deals  only  with  the  form  and  structure  of  the  delivery 
of  logistics  technical  data.  There  is  little  written  about  the  content  or  functionality 
of  the  devices,  and  almost  no  mention  is  made  of  IETMs’  role  in  training. 

Figure  2-1.  IETMs  Guidance 

*DoD  5000.2-R 

•  ADLI SCORM 

•  MIL-PRF-87268 

•  MIL-PRF-87269 


•  SECNAVINST  5000.2B 

•  ASN(RDA)  Memo  (Nov  99) 


•  DRAFT  IETM  Process  Plan 

*  Trammg/IETM  Interface  Guide 


At  the  DoD  level,  there  is  a  single  paragraph  in  the  capstone  acquisition  regula¬ 
tion  that  directs  acquisition  program  managers  to  obtain  logistics  and  maintenance 
documentation  in  digital  form  only  if  it  makes  economic  sense.  The  Department 
also  has  promulgated  two  performance  specifications  that  establish  minimum 
standards  for  presentation  of  maintenance  and  logistics  information.  DoD  also 
chairs  a  cross-service  working  group  that  promotes  collaboration  and  cooperation 
in  the  development  of  distributed  learning  technologies  throughout  DoD.  That 
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working  group  has  developed  the  Advanced  Distributed  Learning  Initiative 
(ADLI)  Shareable  Courseware  Object  Reference  Model  (SCORM),  which  con¬ 
tains  suggested  standards  for  digital  learning  objects. 

The  Navy  has  incorporated  language  into  its  acquisition  regulation  that  simply 
directs  Navy  Program  Managers  (PMs)  to  follow  the  guidance  in  the  DoD  regula¬ 
tion.  The  Assistant  Secretary  of  the  Navy  for  Research,  Development,  and  Acqui¬ 
sition  (ASN[RDA])  has  signed  a  policy  memorandum  that— in  addition  to 
directing  PMs  to  follow  the  guidance  in  DoD  and  Navy  regulations— addresses 
standards  for  delivery  of  digital  data.1  There  is  no  mention  of  training  require¬ 
ments  in  any  of  these  publications,  however. 

The  Navy  developed  the  Draft  IETM  Process  Plan  (IETMPP),  ostensibly  to  en¬ 
sure  that  all  programs  use  a  comprehensive  approach  and  standard  methodology 
to  create  or  convert  to  digital  TMs.  The  IETMPP,  however,  allows  programs  to 
flexibly  determine  an  appropriate  IETM  strategy,  given  their  respective  program 
requirements,  while  ensuring  standard  processes  that  enable  the  DoD  to  continue 
to  support  these  new  logistic  products  using  the  centralized  logistic  management 
systems  currently  in  place.”  2  Thus,  though  the  IETMPP’ s  intent  clearly  is  reason¬ 
able,  this  document  does  not  establish  IETM  development  rules  to  which  acquisi¬ 
tion  programs  must  adhere. 

The  one  document  we  found  that  addresses  training  to  any  significant  degree  is 
the  N7-produced  Training/IETM  Interface  Guide  (T/HG).3  The  T/HG  advises  lo¬ 
gistic  element  managers  and  technicians  on  training  tools,  including  their  imple¬ 
mentation  processes,  and  on  items  that  can  affect  IETM  development.  As  its  name 
implies,  however,  the  T/HG  is  only  a  guide;  it  does  not  carry  directive  authority 
over  acquisition  programs. 

The  IETMPP  and  the  T/HG  are  designed  to  be  complementary.  The  IETMPP 
takes  precedence.  Executive  authority  for  life-cycle  management  of  the  IETMPP 
is  delegated  to  the  Navy  Technical  Manual  Working  Group  via  OPNAV  N4. 

T/HG  life-cycle  management  is  delegated  to  the  Naval  Air  Warfare  Center- 
Training  Systems  Division  (NAWC-TSD)  via  OPNAV  N7. 

Although  the  N7  experts  we  interviewed  indicated  that  proponency  for  IETMs 
rests  with  the  Navy  logisticians  (N4),  we  were  not  able  to  find  an  N4  office  that 
claims  any  directive  authority  with  regard  to  the  acquisition  of  IETMs.  Moreover, 
the  program  offices  we  surveyed  were  only  generally  aware  of  the  foregoing 
guidance  documents.  Also,  it  is  clear  that  logisticians  who  have  primacy  for 
IETM  development  and  the  training  planners  in  N7  work  in  separate  and  distinct 
“stovepipes.”  The  opportunities  for  N7  to  influence  IETM  acquisition  strategy  are 
limited. 

1  ASN(RDA),  Memorandum  for  Distribution,  DON  Policy  on  Digital  Logistics  Technical 
Data,  2  November  1999. 

2  Draft,  Interactive  Electronic  Technical  Manual  (IETM)  Process  Plan. 

3  Chief  of  Naval  Operations  (N7),  Training/IETM  Interface  Guide,  1999. 
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Assessment 

Except  for  the  mandate  to  move  away  from  paper-based  technical  information  to 
information  systems-based  manuals,  the  Navy’s  guidance  regarding  what  IETM 
end  products  should  be  is  quite  general.  Most  current  development  and  acquisition 
activity  is  centered  around  the  effort  to  convert  paper  manuals  for  legacy  systems 
into  electronic  manuals.  This  effort  is  meeting  with  varying  degrees  of  difficulty. 
In  some  instances,  the  nature  of  the  legacy  manuals  imposes  limitations  on  the 
work  that  can  be  performed — some  data  simply  do  not  lend  themselves  to  sophis¬ 
ticated  conversion.  Updating  the  information  before  realistic  SGML  or  XML  con¬ 
version  could  be  conducted  would  take  considerable  time  and  money. 

Nevertheless,  there  is  general  understanding  that  all  TMs  eventually  must  be  put 
into  digitized  formats  that  allow  for  some  interactivity.  There  are  external  stan¬ 
dards — mainly  commercial  or  industry-driven — to  which  many  of  the  develop¬ 
mental  IETMs  are  being  engineered.  At  the  same  time,  the  standards  for  the 
presentation  of  maintenance/logistical  information  are  converging  with  those  for 
the  production  of  ICW.  This  convergence,  however,  seems  to  be  a  function  of 
technology  evolution  rather  than  any  intentional  migration  by  IETM  developers. 

IETM  Acquisition  Management 

The  acquisition  of  IETMs  is  largely  a  bottom-fed  process,  with  requirements  de¬ 
veloped  at  the  program  level  (see  Figure  2-2).  Acquisition  program  offices  are 
nearly  autonomous  and  have  freedom  to  acquire  IETMs  as  they  choose.  (The  pro¬ 
grams,  influenced  by  their  specific  contractor  situations,  are  pursuing  widely  dif¬ 
fering  solutions.)  Thus,  although  many  personnel  at  the  Systems  Command 
(SYSCOM)  and  higher  levels  see  the  need  for  standardization  and 
interoperability,  there  are  limited  incentives  for  the  programs  to  push  very  hard 
toward  that  goal. 
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Figure  2-2.  IETM  Acquisition  Management 


Regarding  acquisition  guidance,  there  is  some  documentation  governing  acquisi¬ 
tion  of  IETMs,  but  it  is  mostly  suggestive  of  how  the  devices  should  be  bought. 
Although  there  is  a  “Technical  Manual  Acquisition  Strategy”  component  in  the 
DoD  acquisition  documentation,  that  handbook  provides  only  guidance  for  pro¬ 
gram  managers  on  how  to  contract  for  and  price  the  new  digital  technical  data. 

The  Secretary  of  the  Navy  (SECNAV)  memorandum,  the  IETMPP,  and  the  T/HG 
each  contain  guidance  for  acquisition  program  managers.  None  of  these  docu¬ 
ments,  however,  actually  defines  a  Navy-wide  strategy  for  development  and  ac¬ 
quisition.  The  focus  of  these  publications  is  the  conversion  of  existing  paper 
maintenance  manuals  into  digital  formats.  The  stovepipe  culture  is  clearly  evident 
at  the  SYSCOM  level  also. 

Assessment 

Navy  acquisition  of  IETMs  is  driven  by  maintenance  and  logistics  considerations 
as  opposed  to  training  issues.  Imposing  widely  accepted  and  consistent  training 
influence  will  be  a  challenge.  Technical  manuals,  which  are  targeted  mainly  to 
maintenance  and  logistics  purposes,  are  procured  by  the  acquisition/logistics 
community.  Training  courseware,  of  course,  is  procured  by  the  training  commu¬ 
nity.  The  two  rarely  interact.  The  “rice  bowl”  culture  must  be  addressed  before 
true  reform  of  this  process  can  take  hold.  Until  then,  realizing  significant  training 
benefit  (or  even  realizing  the  vast  potential  maintenance/logistics  cost-benefit) 
will  be  difficult. 

Summary 

The  Navy’s  IETMs  acquisition  process  is  extremely  decentralized.  Acquisition 
program  offices  are  nearly  autonomous  and  have  the  freedom  to  acquire  IETMs  as 
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they  choose.  The  program  offices,  influenced  by  their  specific  original  equipment 
contractors,  are  pursuing  widely  differing  ETM/IETM  solutions.  While  this  ap¬ 
proach  is  not  bad  in  theory,  in  practice  it  limits  sharing  and  co-use  opportunities. 

Moving  to  a  consistent  electronic  format  for  technical  information  has  been  a  dif¬ 
ficult  process  within  the  Navy.  In  part  this  is  related  to  an  easing  of  standards  un¬ 
der  acquisition  reform  and  the  growing  shifting  of  design  responsibilities  to  the 
contractor.  Furthermore,  because  both  the  logistics  community  and  the  training 
community  have  an  interest  in  the  acquisition  and  implementation  of  lETMs,  the 
Navy  would  probably  benefit  by  establishing  a  lead  agency  to  provide  coordi¬ 
nated  direction  and  guidance. 
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Chapter  3 

IETM  Costs  and  Benefits 


In  this  chapter,  we  examine  the  extent  of  IETM  acquisition  and  use  within  the 
Navy.  We  then  address  IETM  development  and  conversion  processes,  costs,  and 
operational  benefits  that  IETM  users  expect.  We  illuminate  each  of  these  topics 
with  anecdotal  data  from  selected  acquisition  programs. 

IETMs  in  the  Navy 

We  interviewed  each  of  the  SYSCOMs  and  many  of  the  programs  and  venues 
identified  by  SYSCOMs  and  OPNAV  staff  as  having  significant  IETM  programs 
to  determine  the  extent  of  IETM  procurement  by  the  Navy.  Table  3-1  attempts  to 
address  this  question.  The  numbers  do  not  represent  the  totality  of  IETMs  being 
procured  by  the  Navy  but  a  substantial  subset,  based  on  our  interviews. 

Table  3-1.  IETM  Population  by  SYSCOM 


NAVSEA 

NAVAIR 

SPAWAR 

Class  1 

1040 

3882a 

2 

Class  2 

327 

0 

14 

Class  3 

49 

5 

2 

Class  4 

56 

3 

4 

Class  5 

634 

1 

14 

Represents  all  PDF  manuals  on  NATEC  Website;  some  of 
these  manuals  have  Class  2  functionality. 


Naval  Sea  Systems  Command  (NAVSEA)  and  the  Space  and  Naval  Warfare 
Systems  Command  (SPAWAR)  data  were  taken  from  the  Technical  Data  Man¬ 
agement  Information  System  (TDMIS)  managed  by  the  NSWC-Port  Hueneme 
Division  (NSWC-PHD).  Naval  Air  Systems  Command  (NAVAIR)  data  were 
taken  from  the  Naval  Air  Technical  Data  and  Engineering  Service  Command 
(NATEC)  database  managed  by  NATEC  headquarters  at  San  Diego,  CA. 

Clearly  there  are  anomalies  here.  For  instance,  we  do  not  believe  the  Navy  is 
buying  649  Class  5  IETMs.  Based  on  our  discussions  with  NATEC  database  rep¬ 
resentatives,  we  believe  that  the  classes  of  some  of  these  IETMs  are  overstated. 
Some  system  programs  that  claim  to  be  acquiring  Class  5  IETMs  actually  are  get¬ 
ting  Class  4  IETMs,  and  so  on  down  the  line.  Unfortunately,  IETM  descriptions 
are  sufficiently  vague  that  an  accurate  audit  is  impossible.  The  insight  to  be 
gained,  however,  is  that  NAVSEA  and  NAVAIR  appear  to  be  either  doing  the 
minimum  or  going  all  out.  They  are  not  investing  much  in  the  middle-of-the-road 
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capability.  In  general,  the  newest  NAVAIR  aircraft  platforms  are  acquiring  at 
least  Class  4,  whereas  the  vast  number  of  manuals  for  legacy  aircraft  are  being 
converted  to  Class  1/2.  The  SPAWAR  data  probably  are  insufficient  to  enable  us 
to  draw  a  good  conclusion. 

IETM  Characteristics,  Costs,  and  Benefits 

IETM  Desired  Characteristics 

The  introduction  to  IETMs  at  the  end  of  Chapter  1  shows  the  range  of  capabilities 
IETMs  may  possess  and  explains  the  relative  benefits  of  each.  What  are  equip¬ 
ment  developers  looking  for  in  an  IETM?  Our  interviews  with  stakeholders  on  the 
Navy  staff  and  at  the  SYSCOMs  resulted  in  the  following  list  of  desired  features 
for  IETMs: 

♦  Interactivity 

♦  Friendly  interface  (“look  and  feel”  qualities) 

♦  “Smart”  (guides  user  through  procedures) 

♦  User/leamer-centered  (“tailorable”  to  the  user) 

♦  Hierarchical,  revisable  database  (for  flexibility) 

♦  Automated  access,  searchable  “tagged”  content 

♦  Technical  standards  that  allow  reuse 

>  Authoring  in  SGML  or  Extensible  Markup  Language  (XML) 

>-  Data  formats  not  tied  to  display  formats 
>-  Multifunction  display 

♦  Nonproprietary  viewing  software. 

For  maintenance  job  aiding  and  for  training  functionality,  these  attributes  describe 
a  device  that  tailors  its  output  to  the  user’s  information  requirements  in  a  logical, 
ergonomic  way.  The  device  employs  some  degree  of  artificial  intelligence  to 
make  the  right  information  available  in  the  right  context  at  the  right  time.  It  is  in¬ 
teroperable  with  other  automated  systems  the  user  must  employ  and  can  be  used 
as  effectively  in  an  institutional  training  environment  as  on  the  job  site. 
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Acquisition  and  Conversion  Processes 

The  IETMPP  developed  by  NAVSEA  contains  considerable  guidance  on  the  steps 
required  to  acquire  or  convert  technical  data.  The  plan  captures  the  relative  diffi¬ 
culty  of  doing  each  task,  as  well  as  the  pros  and  cons,  in  great  detail.  The  follow¬ 
ing  paragraphs  summarize  the  major  points  that  program  offices  must  consider. 

TMs  can  be  converted  to  any  class  of  IETM  from  existing  hard-copy  TMs,  or  the 
technical  data  can  be  acquired  as  Class  2  or  higher  source  data  from  the  authoring 
contractor  or  government  activity.  Acquisition  involves  a  slightly  different  deci¬ 
sion-making  process  than  conversion. 

Although  Class  4  and  Class  5  IETMs  provide  significant  savings  in  maintaining 
and  updating  the  data,  the  costs  of  conversion  are  high.  The  high  cost  is  related  to 
reauthoring,  or  redesigning  legacy  data  to  take  advantage  of  advanced  functional¬ 
ity.  The  major  costs  of  conversion  are  in 

♦  developing  the  hierarchical  structure; 

♦  reauthoring  the  legacy  TM  to  prepare  data  for  use  in  a  database; 

♦  selecting  the  level  of  granularity  (or  detail)  for  decomposing  each  section; 

♦  reauthoring  and  cleanup  to  eliminate  repetition  and  redundancy; 

♦  adding  photographs,  animations,  movies,  verbal  instructions,  and  other 
supplemental  enhancements; 

♦  naming  common  and  unique  data  objects  and  linking  them  into  a  logical 
presentation;  and 

♦  validation  of  reauthored  information. 

When  higher-class  IETMs  are  created  from  scratch,  the  level  of  data  indenture 
and  granularity  is  optimized  at  the  lowest  or  smallest  level  (i.e.,  the  step).  For 
conversions,  the  costs  of  driving  all  TM  data  down  to  this  optimal  level  may  be 
prohibitively  high.  This  is  not  the  only  logical  option,  however.  Class  4  and  Class 
5  IETMs  can  be  developed  with  the  objects  that  are  roughly  the  same  size  as 
comparable  objects  in  Class  2  or  Class  3  IETMs  (e.g.,  one  paragraph  or  a  proce¬ 
dure).  By  minimizing  the  handling  of  objects  and  substantially  reducing  the  reau¬ 
thoring  desired  or  required,  conversion  costs  can  be  reduced  to  the  same  range  as 
those  for  Class  3.  This  IETM  would  have  the  presentation  features  found  in  a 
normal  Class  4  or  Class  5  IETM,  but  it  would  not  be  as  robust.  This  compromise 
retains  some  redundant  data,  sacrifices  some  database  maintenance  efficiency, 
requires  more  update  effort,  and  reduces  some  flexibility.  Although  some  of  the 
data  may  always  remain  in  the  initial  conversion  state,  the  program  has  the  option 
of  incrementally  reauthoring  specific  sections  of  the  TM  (e.g.,  troubleshooting) 
down  to  the  appropriate  level  of  indenture  (e.g.,  step). 
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Most  TMs  are  involved  in  or  already  have  completed  major  technical  data  conver¬ 
sion  efforts.  These  efforts  primarily  encompass  conversion  from  hard  copy  (paper 
or  aperture  cards)  to  Class  1  (raster)  or  Class  2  (indexed  PDF)  formats.  Therefore, 
many  future  conversions  may  be  from  one  of  these  formats  rather  than  from  the 
original  data  format.  For  instance,  instead  of  having  to  convert  from  paper  to  a 
Class  4 IETM,  the  data  would  be  converted  from  a  Class  2 IETM  to  a  Class  4 
IETM,  with  an  associated  reduction  in  conversion  cost. 

The  formal  Class  5  IETM  consists  of  a  Class  4  IETM  that  shares  an  integrated 
database  with  other  associated  applications.  Consideration  should  be  given  to  the 
data  configuration  issues  entailed  when  multiple  logistics  databases  are  integrated 
into  a  single  database. 

Acquisition  and  Conversion  Costs 

Obtaining  acquisition  cost  data  for  IETMs  was  far  more  difficult  than  we  had  an¬ 
ticipated.  The  reasons  were  many.  First,  the  total  acquisition  cost  for  an  IETM 
often  is  buried  in  numerous  separate  line  items  within  a  program,  which  typically 
are  too  difficult  to  resolve.  In  other  cases,  the  cost  data  are  doggedly  guarded  by 
the  manufacturer  or  government  program  office,  which  was  unwilling  to  provide 
us  with  an  estimated  cost.  Other  times,  the  data  are  buried  in  Contract  Logistics 
Support  (CLS)  documentation.  Still  other  data  are  unique  and  are  difficult  to 
combine  with  analogous  data  from  similar  systems. 

Nevertheless,  some  cost  insight  can  be  obtained  from  paper-to-electronic  conver¬ 
sion  costs.  We  obtained  conversion  cost  estimates  for  each  class  of  IETM,  which 
are  shown  in  Table  3-2. 


Table  3-2.  Cost  of  Converting  Paper  Manuals  to  IETMs 


Interactivity 

Description 

Savings 

Cost/page 

Low 

Raster  scanned  with 
indexing 

1 

Weight/space 

$2 

Moderate 

Intelligent 

2 

$2-$10 

Good 

Interactive  to  user 
(printable) 

3 

$1 0-$25 

High  ^ 

Interactive  to  user 
(objects) 

4 

$40-$1 00+ 

Full 

Interactive  to  user  and 
associated  systems 

5 

T raining  fleet/data 
maintenance 

$200 

Source:  Draft  IETM  Process  Plan,  February  1998,  2-39. 
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NAVSEA  Case  Study 

We  found  that  the  costs  listed  in  Table  3-2,  which  were  included  in  the  IETMPP, 
generally  matched  the  costs  experienced  by  the  people  actually  buying  IETMs 
today.  We  also  can  generalize  ETM  costs  from  some  recent  NAVSEA  conversion 
data.2  To  convert  to  raster  (ETM),  NAVSEA  costs  are  about  $0.50  per  page  (for 
the  average  hull,  mechanical,  and  electrical  [HM&E]  book);  if  one  estimates  that 
each  HM&E  book  is  about  100  pages  on  average,  the  cost  to  convert  is  about  $50 
per  book.  A  typical  IETM  may  combine  up  to  100  manuals.  Thus,  a  low-level 
ETM  may  total  about  $5,000  in  conversion  costs. 

NAVSEA  costs  for  converting  from  paper  to  SGML  Class  3  are  about  $12.50  per 
page.  Half  of  that  amount  is  the  cost  of  conversion  from  text  to  SGML,  and  half  is 
for  quality  assurance  (QA)  of  data  files.  Some  legacy  paper  manuals  are  con¬ 
verted  to  higher-level  (Class  3/4)  IETMs,  but  this  type  of  conversion  generally  is 
very  difficult  and  expensive,  particularly  with  regard  to  QA. 

Many  NAVSEA  and  SPAWAR  programs  are  using  Raytheon’s  Advanced  Inte¬ 
grated  Maintenance  Support  System  (AIMSS™)  to  develop  Class  4  and  Class  5 
devices  with  costs  of  $200-$250  per  page-equivalent.  They  report  that  there  are 
heavy  QA  costs  associated  with  this  process. 

NAVSEA  generalizes  that  an  SGML  ETM  costs  about  one  and  one-half  times 
what  buying  paper  manuals  costs  but  that  update  costs  are  about  one-quarter  those 
of  paper  documents.  They  also  conclude  that  the  reduced  update  costs  for  IETMs 
will  result  in  reduced  life-cycle  costs  that  will  enable  programs  to  recoup  their 
investments  over  relatively  short  durations  (2  to  5  years).  Also,  the  IETM  advan¬ 
tage  is  increasing  because  the  costs  for  electronic  manuals  are  decreasing.  (In  fact, 
Microsoft  has  come  out  with  an  “SGML  author”  add-on  for  Word  for  Windows, 
which  should  reduce  the  cost  of  this  authoring  process  further.) 

Operational  Benefits  of  IETMs 

The  feasibility  and  desirability  of  using  IETM  systems  to  support  military  techni¬ 
cians  in  the  operation,  maintenance,  and  logistical  support  of  weapon  systems 
have  been  accepted  by  all  military  services,  although  no  service  has  maintained 
substantial  records  of  benefits  sustained  to  date.  NSWC  collected  anecdotal  evi¬ 
dence  of  maintenance  improvements  resulting  from  the  use  of  IETMs  for  eight 
selected  projects  in  the  mid-1990s.3  The  maintenance-specific  findings  were  that 
the  use  of  IETMs 

♦  reduced  false  alarm  rates, 

~  Correspondence  with  Martin  Cohen,  NAVSEA  Philadelphia,  Technical  Manual  and  Train¬ 
ing  Branch. 

3  Eric  L.  Jorgensen  and  Joseph  L.  Fuller,  Naval  Surface  Warfare  Center,  Carderock  Division, 
Analysis  of  Eight  Navy  Projects  Accruing  the  Training  Benefit  of  the  Accelerated  Use  of  IETMs, 
21  March  1996. 
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♦  increased  successful  fault  isolations, 

♦  reduced  fault  isolation  times, 

♦  reduced  false  removal  rates, 

♦  increased  the  effectiveness  of  inexperienced  technicians, 

♦  improved  personnel  and  equipment  safety, 

♦  reduced  cycle  time  for  technical  manual  deficiency  reports  (TMDRs),  and 

♦  reduced  technician  time  for  maintenance  reporting  and  parts  ordering. 

The  F/A-18  program  has  made  the  argument  for  IETMs  on  the  basis  of  expected 
reduction  in  false  removal  rates  of  good  components.  False  removals  of  compo¬ 
nents  thought  to  be  faulty— but  eventually  found  to  be  good— were  such  a  prob¬ 
lem  that  a  reduction  of  less  than  half  this  rate  in  the  short  run  paid  for  the 
acquisition  of  IETMs.  Researchers  at  NSWC-CD  found  some  supporting  evi¬ 
dence.4  Their  study,  performed  in  the  early  1990s,  compared  two  groups  of  air¬ 
craft  technicians  performing  fault-isolation  tasks.  One  group  used  conventional 
paper-based  technical  manuals  and  the  other  an  EETM  displayed  on  a  portable 
computer.  The  test  results  showed  considerable  reduction  in  performance  times 
for  complex,  multiple-fault  isolations  and  reduction  of  maintenance  errors  when 
technicians  used  the  IETM/portable  computer  combination. 

In  another  study,  maintenance  technicians  using  a  computer-based  electronic 
document  with  artificial  intelligence  assistance  on  troubleshooting  completed 
tasks  in  less  than  half  the  time  personnel  using  paper  manuals  required.  Novices 
using  the  electronic  job  aid  could  troubleshoot  12  percent  faster  than  experienced 
technicians  using  paper  manuals.5 

IETMs  also  have  the  potential  to  affect  training.  During  limited  pilot  projects 
conducted  in  the  mid-1990s,  for  example.  Navy  researchers  demonstrated  that  the 
use  of  IETMs  could  increase  the  performance  levels  of  inexperienced  technicians 
to  that  of  experienced  technicians  in  complex  tasks  such  as  fault  isolation.  IETMs 
also  can  reduce  the  time  required  to  provide  a  student  with  a  given  level  of  tech- 


4  Mark  T.  Kramer,  Theodore  J.  Post,  and  Samuel  C.  Rainey,  Naval  Surface  Warfare  Center, 
Carderock  Division,  Results  of  a  Joint  Navy/Air  Force  Operational  Test  to  Evaluate  USAF  Inte¬ 
grated  Maintenance  Information  Systems  (IMIS)  Interactive  Electronic  Technical  Manual  (IETM) 
Technology  Applied  to  the  F/A-18  Aircraft,  14  June  1993. 

5  W.  A.  Nugent,  S.  A.  Sander,  D.  M.  Johnson,  and  R.  J.  Smillie,  Troubleshooting  Perform¬ 
ance  Using  Paper  and  Electronic  Documentation,  Naval  Personnel  Research  and  Development 
Center,  Technical  Note  87-41, 1987. 
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nical  competence.6  We  address  the  costs  and  benefits  of  using  IETMs  in  Navy 
training  in  Chapter  4  of  this  report. 

Summary  of  Findings 

Costs  unique  to  IETMs  are  added  to  the  costs  of  acquiring  equipment  technical 
information.  The  true  costs  of  IETMs  begin  with  the  design  (or  authoring)  of  the 
digital  functionality  for  accessing  and  using  the  technical  information.  This  analy¬ 
sis  implies  a  level  playing  field  for  comparison  with  paper  manuals.  It  does  not 
make  allowances  for  the  possibility  that,  conceptually,  there  may  be  a  difference 
in  the  cost  of  supplying  data  for  digital  authoring  versus  supplying  data  in  a  form 
that  will  allow  for  reading  into  linear  files  and  formatting  for  eventual  printing  on 
paper. 

Most  IETM-specific  development  costs  relate  to  software  development.  Interest¬ 
ingly,  the  costs  at  this  stage  have  less  to  do  with  the  volume  of  information  and 
likely  are  in  proportion  to  the  complexity  of  the  item  and  the  level  of  interactivity 
and  functionality  the  user  desires.  The  actual  conversion  or  coding,  however,  does 
incur  a  value-driven  cost  that  is  similar  to  that  shown  in  the  matrix  in  Table  3-2. 
The  higher  the  IETM  class,  the  more  expensive  the  development  alternative. 

Naturally,  there  are  publication  and  distribution  costs  for  printing  and  mailing 
compact  discs  or  authoring  and  posting  Web-based  products.  This  “fielding”  cost, 
which  is  significant  for  paper  manuals,  typically  is  small  in  relation  to  develop¬ 
ment  cost  for  IETMs.  The  larger  the  quantity  of  manuals  required,  the  larger  the 
IETM  production  and  distribution  advantage. 

The  IETM  display  chosen  also  may  add  significant  cost.  If  the  IETM  uses  stan¬ 
dard  PC  platforms  that  are  resident  at  most  schools  and  units,  little  or  no  addi¬ 
tional  investment  may  be  required.  On  the  other  hand,  if  the  IETM  includes  the 
display  device,  the  choice  ranges  from  a  commercially  available.  Internet-ready 
computer  at  about  $500  each  to  military-hardened  Portable  Electronic  Display 
Devices  (PEDDs)  that  can  cost  tens  of  thousands  of  dollars.  Typically,  PEDD 
costs  vary  by  capability  and  quantity  procured. 

Finally,  there  are  life-cycle  support  costs.  These  recurring  costs  apply  regardless 
of  whether  the  IETM  is  used  in  a  training  or  operational  environment.  The  one 
recurring  cost  that  is  highlighted  most  often  as  a  benefit  of  IETMs  over  paper 
manuals  is  the  cost  of  performing  technical  manual  updates  and  changes.  Navy 
organizations  could  save  significant  time  and  money  posting  and  implementing 
updates  and  changes  with  IETMs  than  with  paper  manuals.  The  costs  of  updates 
vary  widely,  depending  mainly  on  the  frequency  and  complexity  of  the  updates 
and  the  means  of  distributing  the  changes.  Personnel  with  whom  we  spoke  agree 


6  Eric  L.  Jorgensen,  Joseph  J.  Fuller,  Theodore  J.  Post,  and  Samuel  C.  Rainey,  NSWC 
Carderock  Division,  Potential  Benefits  to  Navy  Training  Programs  Resulting  from  Increased  Use 
of  Interactive  Electronic  Technical  Manuals,  October  1995. 
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with  this  projection  but  were  unable  to  produce  data  to  provide  a  magnitude  to  the 
projected  savings. 


Insights 

We  found  several  themes  on  a  recurring  basis  during  our  investigation.  First,  de¬ 
spite  the  inertia  of  a  system  that  has  long  relied  on  paper  manuals,  there  is  a  con¬ 
certed  movement  by  all  acquisition  and  logistics  communities  to  acquire  technical 
data  in  digital  form  or  to  convert  paper  manuals  for  electronic  display.  The  design 
of  resulting  IETMs  is  almost  solely  dependent  on  program  finances  or  on  opera¬ 
tional  concerns,  however.  Training  and  logistics  planners  generally  do  not  inte¬ 
grate  their  efforts. 

Many  new  IETMs  require  proprietary  viewers,  making  them  expensive  to  acquire 
and  maintain  and  limiting  their  co-use  possibilities.  For  example,  the  three  major 
aircraft  programs  we  researched  all  experienced  some  problems  stemming  from 
the  use  of  proprietary  elements  in  their  IETMs.  Also,  the  Gas  Turbine  Main- 
tainer’s  (GS)  course  at  Naval  Training  Center  (NTC)  Great  Lakes  had  an  experi¬ 
ence  that  demonstrates  the  dangers  of  using  proprietary  material.  The  Chief  of 
Naval  Education  and  Training  (CNET)  contracted  with  an  original  equipment 
manufacturer  to  provide  a  training  package  to  support  the  gas  turbine  engine 
course.  The  contractor  modified  a  nonproprietary  IETM  and  overlaid  a  proprie¬ 
tary  training  package.  Now,  neither  the  IETM  used  by  the  schoolhouse  nor  the 
training  package  can  be  modified  without  a  significant  outlay  of  funds.  The  pack¬ 
age  has  been  unmodified  since  delivery  in  1994.  Not  surprisingly,  this  situation 
affects  the  quality  of  training  that  GS  school  students  receive  because  they  are 
using  a  somewhat  outdated  training  tool. 

The  Vertical  Takeoff  and  Landing  Unmanned  Aerial  Vehicle  (VTUAV),  on  the 
other  hand,  is  a  blend  of  original  and  legacy  technical  manual  development  whose 
contract  requirement  specifies  nonproprietary  software,  with  heavy  reliance  on 
commercial-off-the-shelf  (COTS)  technology.  The  Acoustic  Research  COTS  In¬ 
tegration  (ARCI)  program  may  be  considered  a  model  program.  The  govem- 
ment/contractor  team  is  developing  a  Class  4  IETM  that  uses  nonproprietary 
software.  Operations  and  training  are  being  designed  integrally,  and  the  IETM  is 
being  used  in  training  as  part  of  an  AEC.  The  devices  will  be  reusable  among 
many  systems  (although  with  the  ARCI  program,  the  IETM  cannot  be  updated  via 
the  Web  because  of  security  concerns).  The  unfortunate  finding,  however,  is  that 
programs  like  ARCI  are  largely  the  result  of  individual  program  office  initiative. 
There  do  not  appear  to  be  the  incentives  in  place  to  foster  proliferation  of  this 
model. 

Summary 

The  SYSCOMs  and  program  offices  differ  with  regard  to  how  they  count  IETMs 
and  with  regard  to  the  categorization  of  the  various  IETM  classes.  This  situation 
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has  led  to  large  accounting  discrepancies.  Consequently,  performing  an  accurate 
audit  of  the  Navy’s  total  cost  in  acquiring  IETMs  is  difficult.  The  Navy  can  rec¬ 
tify  this  problem  partly  by  adopting  a  directive  that  states  a  clear  definition  of  an 
IETM  and  the  various  classes  of  IETMs  and  ensuring  that  the  SYSCOMs  and 
program  offices  understand  and  use  these  definitions. 

Furthermore,  SYSCOMs  do  not  collect  cost  data  associated  with  IETM  develop¬ 
ment,  acquisition,  or  conversion.  If  the  Navy  truly  wants  to  understand  the  extent 
of  its  IETM  investment,  it  will  have  to  encourage  and  possibly  direct  a  cost  re¬ 
porting  framework  and  require  program  offices  to  collect  this  type  of  data.  Bene¬ 
fits,  whether  operational  or  training,  will  remain  meaningless  without  a  clear 
understanding  of  their  associated  costs. 

Although  we  did  not  explicitly  examine  the  operational  benefits  of  using  IETMs, 
there  is  some  indication  from  earlier  work  that  the  benefits  can  be  significant. 
These  benefits  might  include  reduced  repair  times  and  fewer  false  removals.  Col¬ 
lecting  this  type  of  data,  however,  requires  a  concerted  effort  on  the  part  of  the 
operational  community. 

Resource  sponsors  and  decision  makers  who  are  considering  whether  to  resource 
the  acquisition  of  technologies  such  as  IETMs  often  will  establish  return  on  in¬ 
vestment  (ROI)  goals  as  decision  criteria.  IETMs  have  no  monetary  returns  in  and 
of  themselves;  they  claim  all  of  their  benefits  through  life-cycle  cost  savings.7 
Therefore,  the  operational  benefits  attributed  to  the  use  of  IETMs  also  must  be 
converted  into  monetary  returns  or  savings  so  as  to  facilitate  a  robust  benefit 
analysis.  To  date,  very  little  experience  data  of  this  type  have  been  collected  with 
which  to  provide  solid  operational  insights. 


7  James  R.  Carlberg,  Raymond  P.  LeBeau,  Theodore  J.  Post,  Steve  Connell,  and  Mark  Ar¬ 
mijo,  Aviation  Maintenance  Environment  ( AME )  Cost  Benefit  Report,  NSWC  Carderock  Division, 
May  1998. 
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IETMs  and  Navy  Training 


General 

People  who  plan  and  resource  training  generally  accept  the  notion  that  training 
technology  can  positively  affect  training  outcomes.  The  roles  and  impacts  of  vari¬ 
ous  training  technologies  have  been  the  subject  of  many  recent  studies.  The  DoD 
Total  Force  Advanced  Distance  Learning  Action  Team  (TFDLAT)  collected  the 
results  of  several  industry  and  academia-produced  studies  that  attribute  impres¬ 
sive  levels  of  cost  reduction,  training  time  reduction,  and  skills  enhancement  to 
situations  in  which  students  are  “technology-assisted”  (see  Figure  4-1).  This  as¬ 
sistance  ranges  from  Video  Teletraining  (VTT)  to  Web-based  interactive  multi- 
media  instruction  (IMI).  On  the  basis  of  this  research,  the  TFDLAT  concluded 
that  technology-assisted  training  reduces  training  costs  by  30—60  percent;  it  also 
either  reduces  training  time  30  percent  or  increases  post-training  skills  by  about 
30  percent. 

Figure  4-1.  Some  Effect  Sizes  for  Technology-Based  Instruction 


Based  Multimedia  Tutoring  Intelligent 

Instruction  Instruction  Systems  Tutors 

(233  Studies)  (47  Studies)  (11  Studies)  (5  Studies) 


Source:  DoD  TFDLAT  Web  site:  <www.adlnet.org/TFDLAT.htm>. 


Navy  leadership  believes  that  there  are  opportunities  for  resource  savings  in  its 
individual  skills  training  domain  through  prudent  application  of  training  technol¬ 
ogy.  Acting  on  that  belief,  the  Navy  has  begun  an  aggressive  classroom  moderni¬ 
zation  program  it  calls  “training  reengineering.”  The  heart  of  the  reengineering 
program  is  a  suite  of  automation  aimed  at  enhancing  training  and  learning  experi- 
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ences.  IETMs  usually  are  deployed  with  that  automation  suite.  In  fact,  for  techni¬ 
cal  training,  IETMs  represent  the  key  source  of  data  required  to  implement  the 
classroom  reengineering. 

The  Navy  presumes  that  IETMs  will 

♦  enhance  classroom  training  for  technical  specialties,  allowing  the  Navy  to 
reduce  or  eliminate  schoolhouse  training; 

♦  support  job-site  training  by  giving  trainees  access  to  training  resources  any 
place,  at  any  time;  and 

♦  encourage  the  definition  and  development  of  a  shared  IETM  and  training 
database. 

In  fact,  there  is  a  body  of  research  on  benefits  to  the  training  community  of  incor¬ 
porating  training  functionality  into  IETMs.  NSWC-CD  performed  much  of  that 
research  before  1997.  This  research  suggests  that  if  IETM  planning  and  develop¬ 
ment  fully  integrates  the  needs  of  the  training,  maintenance/logistics,  and  opera¬ 
tional  domains,  each  domain  can  realize  noteworthy  gains. 

For  the  training  community,  the  potential  benefit  and  cost  savings  could  be  tre¬ 
mendous.  Specifically,  training  functions  could  be  performed  at  the  job  site, 
which  would  obviate  the  need  for  dedicated  institutional  training  for  those  func¬ 
tions.  The  graphic  in  Figure  4-2  implies  that  some  or  all  of  the  C-school  curricu¬ 
lum  for  some  specialties  could  be  eliminated  with  the  prudent  use  of  sophisticated 
devices. 


Figure  4-2.  Potential  IETM  Classroom  and  Job  Site  Impacts 


Spectrum  from  Classroom  Training  to  Job  Aiding  Tools  in  the  Fleet 


Source:  E.  L.  Jorgensen  and  J.  J.  Fuller,  New  Approaches  for  Navy  Technical  Training  and  Job 
Performance  Aiding  using  Expanded  IETM  Technology,  Carderock  Division  Naval  Surface  Warfare 
Center,  October  1 996,  8. 
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Navy  Individual  Skills  Training 


Training  Process 

The  Navy’s  typical  pattern  for  schooling  of  technical  disciplines  is  shown  in  the 
flow  chart  in  Figure  4-3. 

Figure  4-3.  Typical  Navy  Training  Approach 


The  trainee  is  awarded  a  specialty  and  attends  an  A-school  to  receive  the  basic 
skills  required  to  perform  that  specialty.  The  trainee  then  attends  a  C-school  to  be 
trained  on  higher-order  skills  required  to  perform  diagnostics  and  repair  of  so¬ 
phisticated  equipment.  At  the  C-school,  the  trainee  is  certified  “proficient”  (be¬ 
yond  an  apprentice  level)  to  operate  and  maintain  the  equipment  to  which  he  or 
she  will  be  assigned. 

A  “new”  approach  shown  in  Figure  4 -4 — suggested  by  research  conducted  by 
NSWC — contends  that  effective  use  of  IETMs  in  the  classroom  could  enable  the 
training  base  to  train  selected  students  on  basic  skills  as  well  as  some  advanced 
techniques  in  one  setting.  This  approach  would  obviate  the  need  for  some  or  all 
C-school  training  for  affected  specialties. 

Figure  4-4.  IETM-Enhanced  Training  Approach 


The  expanded  skills  training  would  not  attempt  to  combine  all  of  the  A-  and  C- 
school  training.  Instead,  it  would  train  only  the  most  common  or  frequently  used 
techniques  and  information.  Less-common  or  less  frequently  encountered  situa¬ 
tions  would  be  learned  on  an  as-needed  basis  at  the  technician’s  operational  as¬ 
signment  (i.e.,  the  Fleet).  This  on-the-job  training  (OJT)  could  be  conducted  via 
self-paced  exercises  by  the  technician  or  as  formal  OJT  monitored  by  a  unit 
trainer.  The  trainer  would  use  the  embedded  interactive  courseware  component  of 
an  enhanced  training-  or  performance-aiding  IETM. 

We  were  unable  to  uncover  any  experiential  data  to  suggest  whether  the  proposed 
Carderock  model  would  be  feasible  or  infeasible.  Certainly,  some  Naval  Enlisted 
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Classifications  (NECs)— for  example,  information  technology  (IT)— have  com¬ 
mercial  or  civilian  equivalents,  and  courseware  may  be  bought  off  the  shelf  to  re¬ 
place  some  classroom  training.  Many  military  specialties,  however  such  as  fire 
control  specialists  (FCs) — have  no  civilian  counterparts,  and  the  military  would 
have  to  invest  heavily  to  create  self-paced  courseware.  In  fact,  advances  in  in¬ 
structional  technology  probably  would  have  to  occur  so  that  high-level  simula¬ 
tions  and  feedback  could  be  incorporated  into  such  a  device  economically. 

Trainers  we  interviewed  expressed  understandable  concern  over  the  alternate  ap¬ 
proach.  Their  primary  concern  is  that  pushing  C-school  training  functions  to  the 
Fleet  would  limit  the  quality  and  depth  of  training  students  receive.  Much  of  the 
C-school  curriculum — as  we  observed  in  the  GS  course  addresses  concepts  at  a 
theoretical  level  that  provides  the  breadth  necessary  for  these  technicians-in- 
training  to  think  “outside  the  box”  when  they  return  to  the  Fleet.  In  addition, 
pushing  C-school  responsibilities  on  the  Fleet  will  likely  affect  the  operational 
performance  of  the  ship  as  junior  and  senior  sailors  attempt  to  balance  training 
and  operational  responsibilities.  Moreover,  qualified  instructors  for  each  such 
training  requirement  may  have  to  be  detailed  to  the  Fleet. 

Each  of  these  developments  would  run  counter  to  the  optimum  manning  paradigm 
of  the  SMARTSHIP  concept  or  the  “zero-based”  crewing  concept  of  the  DD-21. 
These  new  concepts  call  for  limited  apprenticeship  opportunities.  NTC  Great 
Lakes  instructors  recommend  limiting  the  amount  of  training  onboard  ship  to  re¬ 
fresher  training  only. 

CNET  has  suggested  a  second  alternative  to  the  base  model  (Figure  4-5).  The 
CNET  model  further  leverages  the  capabilities  of  job  performance-aiding  devices. 
In  fact,  the  device  in  the  CNET  model  would  be  called  an  Interactive  Electronic 
Training  and  Technical  Manual  (IETTM).  Developed  with  training  requirements 
assuming  as  much  importance  as  technical  functionality,  this  device  would  have 
several  levels  of  interactive  courseware  built  in. 

Figure  4-5.  CNET  Suggested  Training  Model 


The  CNET  model  calls  for  reducing  the  A-school  content  to  be  more  easily  man¬ 
aged  by  the  young  sailor.  Many  lower-density  duties  and  functions  would  be  in¬ 
cluded  in  the  enhanced  IETTM,  and  the  sailor  would  learn  them  by  using  a  self- 
paced,  just-in-time  method  during  his  or  her  Fleet  tour.  As  the  sailor  advances,  he 

1  DD-21  Web  site  at  <http://sc21.crane.navy.mil/dd21/program/overview.htm>. 

2  Interview  with  Terry  Halvorsen,  CNET,  15  August  2000. 
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or  she  would  attend  a  reduced-scope  C-school  and  a  “Master-level”  school,  both 
at  the  Fleet  Concentration  Area.  The  IETTM  would  recognize  the  increased  cog¬ 
nitive  capabilities  of  the  user  and  structure  further  self-paced  training  and  per¬ 
formance  aiding  to  match  the  user’s  requirements. 

Savings  in  this  model  relative  to  the  current  approach  would  come  from  the  re¬ 
duced  length  of  the  A-school  and  the  reduced  size  and  greatly  reduced  temporary 
assigned  duty  (TAD)  requirements  of  the  Fleet  C-schools.  CNET  contends  that, 
unlike  the  Carderock  model,  this  model  is  gaining  some  support  from  the  Fleets 
and  the  training  community. 

The  CNET  approach  holds  some  intriguing  possibilities.  It  clearly  assumes  that 
the  training  and  education  community  could  reconfigure  to  support  the  increased 
number  of  Fleet  C-schools  and  that  the  acquisition  and  training  communities 
could  work  out  some  complicated  collaboration  and  funding  issues.  It  also  places 
heavy  reliance  on  having  a  robust  job  aid  that  supplements  classroom  training. 
This  model  is  worth  exploring. 

Navy  Training  Technology 

The  Navy’s  training  reengineering  program  seeks  to  reduce  the  time  and  re¬ 
sources  required  to  conduct  skills  training  by  infusing  training  technology  into  the 
classroom  and  by  updating  training  curricula  to  maximize  the  effectiveness  of  the 
new  technology.  The  basic  classroom  reengineering  piece  consists  of  AECs, 
Learning  Resource  Centers  (LRCs),  and  Interactive  Multisensor  Analysis  Trainers 
(IMATs).3 

AECs  promise  increased  classroom  efficiency  and  effectiveness  by  making  class¬ 
rooms  interactive,  multi-media  learning  vehicles.  LRCs,  to  be  located  at  selected 
schools  and  training  sites,  provide  alternate  means  for  students  to  access  curricu¬ 
lum  training  materials  and  professional  enhancement  material  in  electronic  for¬ 
mats.4  LRCs  are  intended  to  reduce  training  attrition  and  save  student  billets  by 
moving  students  out  of  formal  training  and  into  the  Fleet  sooner.  IMATs  are  in¬ 
tended  to  make  hands-on  training  more  effective.  The  cumulative  result  should  be 
reduced  training  time  per  student  and  fewer  required  refresher  courses. 

IETMs  and  Training 


The  acquisition/logistics  community  procures  TMs  that  are  targeted  mainly  to 
maintenance  and  logistics  purposes.  Training  courseware,  of  course,  is  procured 
by  the  training  community.  Developing  IETMs  that  will  serve  these  two  ideologi¬ 
cal  masters  is  a  challenge.  Given  the  current  Navy  infrastructure  and  the  way  sup- 


3  Draft  CNET  Instruction,  “Return  on  Investment  (ROI)  for  Training  Reengineering  Invest¬ 
ments,”  1999. 

4  Department  of  the  Navy,  POM-OO  Training  Baseline  Assessment  Memorandum,  23  March 
1998,  Tab  A-2. 
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port  documentation  is  maintained,  IETM  developers  must  provide  a  means  for 
separate  technical  manual  and  training  documentation  deliverables,  if  both  are 
developed  concurrently.  If  they  are  delivered  separately,  developers  should  pro¬ 
vide  a  means  for  updating  them  that  ensures  that  they  will  remain  consistent  with 
one  another.5 

There  are  numerous  examples  of  IETMs  that  have  been  used — in  classroom 
situations  and  outside  them— to  facilitate  technical  training.  Observers  generally 
accept  that  at  least  a  Class  2  or  3 IETM  is  required  to  provide  the  navigation 
functionality  necessary  to  efficiently  train  on  complicated  topics.  For  practical 
exercises  and  low-density  training  events,  an  IETM  with  courseware  integrated 
can  be  used.  Higher-level  IETMs  also  can  be  structured  to  recognize  the  level  of 
the  user  and  tailor  OJT  and  remedial  training  for  the  specific  user,  as  well  as  to 
update  the  user’s  learning  category  as  lessons  are  successfully  negotiated. 

Advocates  assert  that  incorporating  IETMs  into  training  will  result  in  a  reduction 
in  classroom  hours  required  to  conduct  technical  training.  This  result  also  may 
lead  to  other  benefits,  such  as  reductions  in  the  number  of  instructors  required, 
reductions  in  the  number  of  people  in  the  training  pipeline,  and  shorter  pipelines. 
They  contend  also  that  because  IETMs  should  make  technical  training  more  ef¬ 
fective,  grade-point  averages  will  improve  and  attrition  rates  will  go  down. 

Training-IETM  Interface 

To  facilitate  training  with  IETMs,  a  major  interface  between  the  training  devel¬ 
opment  and  the  IETM  development  processes  clearly  must  be  considered.  The 
Navy’s  guide  for  approaching  this  interface  delineates  four  strategies: 

♦  Coordinated  (exchanging  files,  interfacing  software)— independent  IETM 
and  training  product  development  with  planned  and  coordinated  exchange 
of  IETM  files,  software,  and  so  forth  for  interface  with  automated  cur¬ 
riculum  development  programs 

♦  Concurrent  (developing  some  shared  data,  sharing/interfacing  software)— 
simultaneous,  planned,  and  coordinated  development  of  IETM  and  train¬ 
ing  support  products,  particularly  development  and  use  of  shared  data  and 
software;  initial  joint  planning  and  requirements  determination  enable  data 
integration  into  training  curriculum 

♦  Integrated  (developing  a  shared  database;  sharing  software) — structured, 
concurrent,  and  shared  development  of  all  maintenance,  operations,  and 
training  materials  into  a  single,  fully  integrated  database  that  uses  multiple 
tailored- view  packages  to  present  material  to  various  users 


5  DON,  Training/Interactive  Electronic  Technical  Manual  Interface  Guide,  March  2000. 

6  Tri-Service  Technical  Manual  Working  Group  (NTMWG),  Draft  Interactive  Electronic 
Technical  Manual  (IETM)  Process  Plan  (IETMPP),  February  1998,  sec.  8.2.1. 
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♦  Embedded — combining  all  requirements  and  materials  into  an  integrated 
database  with  a  single  view  package  so  that  training  materials  are  indistin¬ 
guishable  from  maintenance,  operations,  and  other  materials,  enabling  op¬ 
erations,  maintenance,  and  training  personnel  to  use  a  single  product. 

There  are  several  advantages  to  interfacing  ICW  with  IETMs.  From  an  opera¬ 
tional/maintenance  perspective,  the  user  can  access  content-specific  instructional 
text,  graphics  (static  or  dynamic/animated),  and/or  video  to  enhance  or  assist  with 
technical  concepts  or  procedures.  From  the  training  perspective,  having  an  inter¬ 
active  electronic  technical  database  available  during  an  ICW-supported  lesson 
makes  possible  access  to  realistic  technical  information  that  is  extensive  in  depth 
and  scope.  All  of  the  necessary  learning  elements  (such  as  menus,  objectives,  and 
embedded  questions)  can  be  resident  in  the  ICW;  specific  technical  data  (text, 
graphics,  and  video)  reside  in  an  underlying  IETM  database.  Tying  ICW  to 
IETMs  also  provides  instructors  with  an  on-time  reference.  The  higher  classes  (4 
and  5)  have  the  greatest  potential  to  affect  how  maintenance  training  takes  place. 

IETM  Costs 

To  get  the  most  from  IETMs  in  the  classroom,  they  must  be  employed  with  other 
AEC  components  and  have  associated  interactive  courseware.  This  fact  compli¬ 
cates  forecasts  of  IETMs’  costs  and  impacts.  If  AECs  are  in  place  for  the  planned 
IETM  application,  the  cost-benefit  ratio  will  be  one  value.  If  not,  the  cost-benefit 
ratio  will  be  a  different — probably  significantly  lower — value.  An  assessment  of 
potential  costs  and  savings  for  IETMs,  however,  should  account  for  AEC  suite 
investment  costs  if  AECs  are  not  already  in  place. 

There  also  are  life-cycle  support  costs  to  consider.  These  recurring  costs  relate  to 
the  use  of  IETMs  in  a  training  environment  or  in  an  operational  environment.  The 
cost  of  performing  technical  manual  updates  and  changes  is  one  recurring  cost 
that  often  is  cited  as  a  benefit  of  IETMs  over  paper  manuals.  Navy  organizations 
may  save  significant  time  and  money  posting  and  implementing  updates  and 
changes  with  IETMs  rather  than  with  paper  manuals.  The  magnitude  of  the  sav¬ 
ings  will  depend  on  the  media  and  method  for  updating  the  IETM,  the  class  of 
IETM,  and  the  number  of  paper  changes/pages  affected,  among  other  factors.  An¬ 
ecdotal  information  from  NAVSEA  sets  the  savings  (or  cost  avoidance)  for  up¬ 
dating  IETMs  at  about  one-fourth  the  cost  of  updating  comparable  paper 
manuals.7 

The  costs  of  updates  vary  widely,  depending  mainly  on  the  frequency  and  com¬ 
plexity  of  the  updates  and  the  means  of  distributing  the  changes. 


7  Martin  Cohen,  NAVSEA  Ships  Systems  Engineering  Station,  Philadelphia,  PA,  14  Septem¬ 
ber  2000. 
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In  addition  to  hardware  and  courseware  acquisition  costs,  there  also  are  costs  for 
converting  the  A-  or  C-school  course  and  then  implementing  reengineered 
courses. 

IETM  Benefits 

The  experts  we  spoke  with  claim  to  have  observed  the  following  major  benefits  of 
using  IETMs: 

♦  Cost  savings  for  printing,  distribution,  maintenance,  and  storage  (as  de¬ 
scribed  in  Chapter  3) 

♦  Reduced  life-cycle  support  costs  related  to  changes  and  updates 

♦  Improved  training  experience  reflected  in  reduced  classroom  time  and  im¬ 
proved  grade-point  averages: 

>  assists  instructional  material  development  process 

>  reduces  TM  navigation  time 

>•  increases  student  comprehension8 

♦  Improved  maintenance  operations  due  to  more  efficient  technicians 

We  reiterate,  however,  that  these  benefits  are  very  difficult  to  verify  empirically 
because  of  the  lack  of  captured  data.  Moreover,  the  IETM  usually  is  introduced 
into  training  at  the  same  time  that  the  school  incorporates  the  AEC.  Trainers  and 
curriculum  developers  also  are  making  compound  changes  as  they  incorporate 
AEC/IETMs  into  their  courses.  The  Sub  School,  for  instance,  in  addition  to  in¬ 
stalling  AECs,  significantly  changed  the  curriculum  for  most  affected  courses.  In 
some  cases,  they  increased  the  course  content  by  an  estimated  30  percent.  Thus, 
isolating  the  change  that  one  may  attribute  to  the  IETM  is  very  difficult. 

Training  institutions  we  researched  are  realizing  training  time  reduction— -but  not 
of  the  magnitudes  some  previous  studies  presumed.  Courses  we  surveyed  did  re¬ 
alize  10—15  percent  time-to-train  reductions  (not  the  30—40  percent  savings  pre¬ 
dicted  by  early  studies).  Also,  instructors  still  must  take  time  to  manually  create 
Microsoft  PowerPoint  presentations  with  which  to  conduct  instruction.  There  are 
no  significant  changes  in  student-to-instructor  ratios  or  graduation  rates— meas¬ 
ures  commonly  referred  to  in  concept  studies. 

The  GS  school,  for  instance,  was  able  to  reduce  course  length  by  about  10-12 
percent  with  the  installation  of  its  AECs.9  Although  this  figure  represents  signifi- 

8  Telephone  interview  with  Lt  Cdr  Brian  Little,  Navy  Sub  School,  New  London,  CT,  20  June 
2000. 

®  Interview  with  Commander  William  Hawn,  Officer-in-Charge  Engineering  Systems 
Schools,  Great  Lakes,  IL,  21  June  2000. 
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cant  savings  if  extrapolated  to  more  and  larger  courses,  it  is  well  short  of  the  ap¬ 
proximately  30  percent  classroom  training  reductions  that  some  of  the  current  lit¬ 
erature  assumes. 

This  reduced  course  time  eventually  should  lead  to  reduced  time  for  students  in 
the  training  pipelines  and  less  TAD  requirement,  but  schools  we  contacted  are  not 
seeing  shorter  pipelines  to  date.  The  staffing  and  personnel  management  impacts 
of  having  IETMs  in  the  schoolhouse  and  at  the  Fleet  also  must  be  evaluated. 

There  may  be  more  efficiencies  and  savings  in  these  domains,  as  well  as  to  the 
Navy  as  an  enterprise,  if  training  and  education  pipelines  shorten  for  some  techni¬ 
cal  specialties. 

Impact  of  IETMs  on  Training 

IETMs  and  Classroom  Training 

Recent  Navy  research  shows  a  trend  of  increasing  demand  for  new  NECs  in  the 
Navy.  A  Navy  survey  of  the  Truman  Battle  Group  revealed  that  more  than  70 
percent  of  new  or  upgraded  systems  deployed  were  supported  by  some  form  of 
training.  About  25  percent  of  these  required  that  either  a  new  NEC  or  an  increase 
in  the  number  of  personnel  holding  current  NECs  be  assigned  onboard.10 

For  each  new  NEC,  the  Navy  must  create,  implement,  and  sustain  an  individual 
skills  training  program,  mainly  in  institutional  settings.  The  cost  to  train  sailors  in 
an  NEC  can  range  from  $5,000  to  $25,000.* 1  If  the  use  of  IETMs  could  replace 
some  of  this  training,  the  Navy  could  save  significant  amounts  of  training  re¬ 
sources. 

CNET  has  compiled  “before-and-after”  time  and  cost  statistics  on  the  AEC  in¬ 
stallations  they  have  made  since  1997.  (Caution:  The  figures  in  the  following  il¬ 
lustrations  include  the  total  time  students  are  charged  against  training  accounts, 
not  just  classroom  training  time.)  Anecdotal  CNET  data  for  electronics  technician 
(NEC  ET)  (communications  and  radar)  A-school  training  indicate  large  reduc¬ 
tions  in  time-to-train:  about  837  man-years  (of  2,710  possible).  The  cost  associ¬ 
ated  with  these  time  savings  is  about  $30  million — or  about  one-third  of  what  was 
traditionally  spent  for  those  courses  before  reengineering.  CNET  estimates  that  on 
average  it  is  realizing  man-year  savings  of  about  8-15  percent  for  courses  like 
these. 

As  we  have  noted,  measurement  of  the  specific  impact  of  IETMs  in  training  set¬ 
tings  is  complicated  by  the  lack  of  available  data  and  the  absence  of  any  clean 
way  to  “back  out”  IETM  effects.  We  can  get  some  insights  into  the  IETM  contri¬ 
bution,  however,  by  evaluating  some  gross  results  from  AEC  deployments.  Fig- 


10  OPNAV  N81,  Assessment  Division  briefing,  “Training  and  Education  IWAR  Individual 
Skills  Assessment  Process,”  5  September  2000. 

11  Interview  with  Maureen  Davidovich,  OPNAV  N7,  10  October  2000. 
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ures  4-6  and  4-7  show  CNET  data  for  reengineered  training  courses— some  of 
which  had  IETMs  incorporated  into  the  AECs  and  some  of  which  did  not. 

Figure  4-6.  ETA-School  Reengineering 
(Average  Days  Saved) 


Figure  4-7.  ETA-School  Reengineering 
(Average  Dollars  Saved ) 


Average  Manyear  Dollars  Reduced 


The  sample  size  that  these  data  reflect  is  very  small  (20  courses),  so  the  statistical 
significance  of  the  results  is  not  very  high.  One  may  conclude,  however,  that  there 
probably  is  a  positive  difference  between  time  and  cost  savings  achieved  in 
courses  with  IETMs  and  those  without  the  devices.  CNET  estimates  that  IETMs 
may  account  for  about  10-15  percent  of  the  8-15  percent  in  training  dollars  saved 
as  a  result  of  installing  AECs.  That  savings  amounts  to  about  $2.5-$3  million  to 
date  for  ET  A-school  courses  alone. 

One  of  the  more  important  aspects  of  the  Navy’s  training  reengineering  strategy  is 
that  it  creates  the  requirement  for  a  new  evaluation  measure  of  effectiveness 
return  on  investment.  Before  making  the  investment  in  training  technology,  the 
Navy  must  make  an  assessment  that  it  will  realize  a  monetary  savings  by  making 
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that  investment.  The  Navy  has  established  an  8  percent  return  in  2  years  as  the 
standard.  CNET  currently  interprets  ROI  to  be  the  net  of  monies  invested  in 
reengineering  and  postulated  savings  from  (presumably)  reducing  training  time 
per  student. 

IETMs  AND  Job-Site  Training 

The  concept  of  incorporating  training  into  IETMs  for  the  benefit  of  the  technician 
at  his  or  her  work  site  was  explored  in  a  recent  test  that  used  aircraft  platforms. 

The  Aviation  Maintenance  Integrated  Diagnostics  Demonstration  (AMIDD) 
tested  the  operational  maintenance  efficiency  effects  of  using  off-aircraft  diag¬ 
nostics  with  computer-aided  maintenance  techniques.  The  test  showed  that  the  use 
of  IETMs  greatly  improved  the  diagnostic  and  maintenance  results  of  the  techni¬ 
cians.  The  testers  also  concluded,  however,  that  integrating  training  into  this  con¬ 
cept  is  important.  They  wrote: 

The  similarities  between  the  data  used  to  develop  Computer  Based 
Training  (CBT)  and  Interactive  Electronic  Technical  Manuals  (IETMs) 
should  allow  for  a  seamless  integration  of  the  two  to  either  share  data 
fields  and  reduce  duplicate  data  or  at  least  linking  the  databases.  This 
will  allow  for  single  data  input  for  training  and  technical  manuals.  This 
will  enhance  training  in  the  classroom  and  in  the  fleet  as  the  maintainer 
will  be  able  to  launch  from  training  applications  into  the  applicable  tech¬ 
nical  manual  or  vice  versa.1- 

According  to  N7,  ICW  and  job-aiding  technologies  are  converging.  With  this  in 
mind,  some  programs  are  approaching  the  issues  of  training  and  mainte¬ 
nance/logistical  support  as  related.  They  foresee  tangible  and  intangible  benefits 
of  fully  integrating  their  plans  for  maintenance  and  logistics  with  their  training 
plans.  One  such  program  is  ARCI.  This  program  and  the  Virginia  Class  Sonar, 
Combat  Control,  Architecture  (S/CC/A)  program  are  building  to  an  Integrated 
Development  Program  (IDP)  that  would  concurrently  develop  training  and  logis¬ 
tics  products  across  both  programs.  The  purpose  is  to  increase  asset  utilization 
and  reduce  costs  for  both  programs.  The  ARCI  team  has  logisticians  and  trainers 
working  together  to  create  IMI  that  will  communicate  directly  with  the  system’s 
IETM  to  provide  “a  seamless  interface  between  the  IETM  and  IMI.” 

When  system  developers  are  designing  new  systems,  they  are  required  to  consider 
the  training  implications  of  the  item  and  even  to  incorporate  embedded  training 
systems,  if  feasible.  The  real  key  appears  to  be  to  minimize  the  human  interface 
requirements  of  the  end  item  design.  Fewer  human  interface  requirements  mean 
fewer  topics  requiring  training.  For  training  requirements  that  remain,  use  as 
smart  a  performance  aid  as  possible.  If  the  IETM  is  designed  with  the  intelligence 


12  J.  R.  Carlberg  et  al.,  Aviation  Maintenance  Integrated  Diagnostics  Demonstration 
(AMIDD)  Test  Report,  Carderock  Division  Naval  Surface  Warfare  Center,  1998, 32. 

13  Lockheed  Martin  Undersea  Systems,  Integrated  Development  Program  Integrated  Support 
Plan,  1  November  1999, 6-5. 
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to  know  what  tasks  the  user  needs  training  on,  it  can  be  used  effectively  within  the 
scope  of  a  planned  training  continuum. 

Summary 

We  found  that  IETMs  are  not  yet  widely  disseminated  in  training  venues.  Where 
they  are  in  classrooms,  IETMs  generally  are  incorporated  into  training  as  part  of  a 
larger  course  reengineering.  This  fact  makes  assessing  their  impact  empirically 
nearly  impossible  without  a  controlled  test  or  pilot. 

Training  technology  suites  are  not  yet  achieving  time  and  cost  savings  in  the 
magnitude  theorized  in  early  studies.  IETMs’  contribution  to  the  realized  savings 
is  probably  about  10  percent.  In  addition  to  providing  training  capability  to  the 
job-site,  IETMs  in  operational  settings  also  results  in  improved  maintenance  op¬ 
erations. 

The  greatest  potential  benefits  for  training  occur  with  devices  that  have  higher 
degrees  of  interface  between  technical  data  and  training  courseware.  Achieving 
these  levels  of  interface  routinely,  however,  probably  requires  an  increased  level 
of  involvement  by  the  Navy’s  acquisition  management  leadership. 
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Chapter  5 

Findings  and  Conclusions 


General 

In  this  chapter,  we  summarize  the  major  findings  and  conclusions  discussed  in  the 
preceding  chapters.  We  have  reduced  the  issues  to  the  four  that  we  believe  pro¬ 
vide  the  most  insight  to  the  Navy.  Where  appropriate,  we  also  suggest  ways  the 
Navy  may  address  these  issues. 

Assessing  IETM  Cost-Benefit 

Findings 


Probably  the  overarching  conclusion  from  this  research  is  that  drawing  conclu¬ 
sions  about  the  Navy’s  costs  and  benefits  that  come  with  using  IETMs  will  be  ex¬ 
tremely  difficult.  An  objective  and  robust  cost-benefit  analysis  is  nearly 
impossible  in  the  short  term,  for  several  reasons. 

First,  it  may  be  too  early  in  the  evolution  of  IETMs  to  draw  meaningful  conclu¬ 
sions  about  their  return  on  investment  or  their  long-term  impact  on  readiness.  A 
good  cost-benefit  analysis  must  be  able  to  extrapolate  IETMs’  experience  in  op¬ 
eration  over  a  “life  cycle.”  Military  schoolhouses  are  only  now  integrating  IETMs 
and  electronic  classrooms  into  how  they  do  business.  These  systems  are  experi¬ 
encing  predictable  growing  pains.  There  also  is  a  lack  of  practical  experience  with 
sufficient  numbers  of  mature,  fielded  systems. 

We  found  that  the  organizations  in  the  best  positions  to  collect  data  concerning 
IETMs  costs  and  benefits  don’t.  Neither  the  Service  School  Command  (SSC)  at 
Great  Lakes  nor  the  Submarine  School  at  New  London  routinely  records  changes 
in  course  length,  student  scores,  or  student  throughput  capability  resulting  from 
the  insertion  of  technology  into  their  classrooms.  SYSCOMs  had  no  operational 
experience  data  on  which  to  base  their  acquisitions. 

The  major  problem  with  obtaining  impact  data  is  that  IETMs  are  almost  never 
deployed  in  isolation.  They  are  fielded  as  part  of  a  suite  of  training  technology  in 
their  incorporation  in  classrooms.  At  operational  sites,  the  introduction  of  IETMs 
often  is  accompanied  by  other  operational  improvements.  Extracting  the  effects 
one  may  attribute  to  the  presence  of  IETMs  is  difficult. 

We  also  encountered  significant  roadblocks  to  efficiently  obtaining  cost  data. 
Sometimes  the  data  are  doggedly  guarded  by  the  manufacturer  or  the  government 
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program  office.  In  other  cases,  the  data  are  buried  in  CLS  documentation.  Be¬ 
cause  of  the  wide  range  of  approaches  to  acquiring  IETMs,  some  data  are  unique 
and  therefore  are  difficult  to  combine  with  analogous  data  from  similar  systems. 

Conclusions 

There  is  a  critical  need  for  cost  and  benefit  data.  To  understand  the  full  potential 
impact  of  IETMs  on  training  and  operations,  the  Navy  will  have  to  be  able  to  say 
where  the  dollars  are  going,  what  it  is  getting  for  those  dollars,  and  what  return  it 
may  expect  for  future  expenditures.  Until  such  empirical  evidence  is  available, 
IETMs’  advantages  or  disadvantages  will  continue  to  be  speculative.  Collecting 
data  and  making  it  available  may  require  flag  officer  involvement  at  the  OPNAV, 
SYSCOM,  and  training  command  levels  to  encourage  the  pertinent  organizations 
to  capture  the  data. 

There  are  many  mechanisms  for  obtaining  usable  data  for  an  analysis.  One  of  the 
best  means  of  assessing  IETMs’  cost/benefit  is  to  conduct  a  scientifically  de¬ 
signed  pilot  project.  Researchers  may  use  two  comparable  classes  and  collect  ex¬ 
perience  and  results  data  as  one  class  trains  with  IETMs  and  the  other  trains 
without  the  devices.  Alternatively,  IETMs  can  be  introduced  into  a  course  for 
which  results  are  already  documented.  In  this  case,  researchers  would  compare 
“before  and  after”  observations.  These  experiments  also  can  be  conducted  at  op¬ 
erational  job  sites. 

IETM  Acquisition  Policies 

Findings 

The  Navy  exercises  little  management  oversight  of  IETM  development  and  acqui¬ 
sition.  The  process  is  decentralized  to  a  fault.  There  is  no  central  guidance  or 
authority  to  which  acquisition  program  offices  can  look  for  the  big  picture.  Po¬ 
tential  efficiencies  and  savings  may  never  be  realized  without  directive  interven¬ 
tion  by  Navy  leadership. 

Although  the  Navy  has  published  general  guidance  that  asks  acquisition  program 
managers  to  acquire  digital  technical  information  where  it  is  deemed  “cost  effec¬ 
tive,”  there  are  no  related  guidance  documents  that  address  how  programs  should 
determine  cost  effectiveness. 

Navy  acquisition  programs  acquire  IETMs  based  almost  solely  on  maintenance 
and  logistical  criteria.  Training  rarely  plays  a  role  in  defining  the  type  and  class  of 
device  a  program  acquires.  Logistics  and  training  stovepipes  exist  that  severely 
limit  the  amount  of  interaction  these  two  communities  exercise. 

The  widespread  use  of  proprietary  software  clearly  hampers  the  cost-effective  ac¬ 
quisition  and  employment  of  IETMs.  A  prime  benefit  of  COTS  technology  is  that 
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systems  may  be  constantly  upgraded  and  improved  at  modest  or  no  cost  to  the 
government.  This  factor,  of  course,  is  in  keeping  with  AEC  imperatives.  All  ac¬ 
quisition  programs  should  acquire  COTS  technology  or  rights  to  IETM  software 
for  the  government. 

Conclusion 

The  Navy  could  benefit  by  establishing  a  “lead”  office — probably  in  the  office  of 
ASN(RDA),  with  representation  from  N4  and  N7,  among  others — to  serve  as  the 
defining  authority  for  IETM  acquisition.  This  authority  must  establish  an  overall 
acquisition  vision  that  breaks  the  “rice  bowl”  mentality  that  permeates  the  current 
process. 

One  of  the  first  tasks  for  the  lead  office  would  be  to  develop  consistent,  measur¬ 
able  definitions  for  IETMs  so  that  all  stakeholders  are  counting  the  same  things. 
The  lead  office  also  would  define  the  cost  effectiveness  metrics  that  PMs  would 
use  to  conduct  their  business  case  analyses.  The  office  must  address  the  issue  of 
proprietary  viewer  software  and  develop  and  enforce  interface  and  functionality 
standards. 

Buying  IETMs 

Finding 


Most  IETM-specific  development  costs  relate  to  software  development.  These 
costs  have  less  to  do  with  the  volume  of  information  and  likely  are  in  proportion 
to  the  complexity  of  the  item  and  the  level  of  interactivity  and  functionality  the 
user  desires.  The  actual  conversion  or  coding,  however,  does  incur  a  value-driven 
cost. 

The  predominant  reason  that  Navy  programs  acquire  IETMs  is  to  save  the  high 
operations  and  support  (O&S)  costs  associated  with  paper  technical  manuals.  In 
general,  programs  are  buying  the  lowest-class  devices,  without  much  analysis  to 
understand  how  matching  the  class  of  IETM  to  the  anticipated  maintenance  or 
training  tasks  may  affect  their  ROI. 

Conclusion 

Regardless  of  whether  the  Navy  wishes  to  assess  the  relative  costs  and  benefits  of 
IETMs  to  address  training  and  education  issues,  there  should  be  a  concerted  effort 
to  put  data  structures  in  place  to  assess  whether  the  Navy  is  spending  its  technol¬ 
ogy  dollars  wisely.  The  Navy  would  benefit  from  a  comprehensive  review  of  its 
acquisition  cost  reporting  requirements,  education  and  training  database  struc¬ 
tures,  and  logistics  data  management  and  reporting  requirements  to  determine  if 
data  capture  capabilities  are  sufficient  to  address  routine  cost-benefit  or  ROI  que¬ 
ries. 
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If  avoiding  the  high  O&S  costs  of  legacy  manuals  is  a  compelling  reason  to  ac¬ 
quire  IETMs,  the  Navy  should  study  ways  to  attack  these  costs  as  an  enterprise. 
This  strategy  would  require  the  involvement  of  all  resource  sponsors  and  claim¬ 
ants  who  work  with  technical  data. 

The  study  must  first  determine  as  many  of  the  factors  that  influence  the  high  O&S 
costs  as  possible.  Then  the  Navy  can  determine  which  of  those  factors  can  be 
controlled  for  the  greatest  O&S  cost/savings  impact.  Alternatives  such  as  direct¬ 
ing  acquisition  and  distribution  of  various  classes  of  IETMs,  investing  in  wearable 
computers,  developing  a  Web-based  repository  of  all  technical  data,  or  other  ap¬ 
proaches  could  be  modeled  to  determine  the  best  corporate  approach. 

IETM  Training  Benefits 

Findings 


Although  there  are  some  observed  benefits  to  using  IETMs  in  training,  users  are 
not  yet  seeing  the  dramatic  time  and  cost  savings  that  earlier  studies  predicted. 
Reengineering  of  classroom  training  is  resulting  in  better  student  performance  and 
some  course  length  reductions.  We  temper  this  finding  with  the  admission  that  we 
cannot  draw  firm  conclusions  about  the  cost-benefit  ratio  or  ROI  of  IETMs  be¬ 
cause  we  cannot  isolate  benefits  specific  to  IETMs. 

Several  technology  initiatives  are  being  promulgated  throughout  the  Navy  simul¬ 
taneously,  but  their  effects  may  be  watered  down  unless  the  initiatives  propo¬ 
nents  are  cognizant  of  one  another  and  all  relevant  parties  are  required  to  comply 
with  certain  imperatives. 

Conclusions 

If  one  logically  assumes  that  there  are  fiscal  and  operational  benefits  to  using 
IETMs  rather  than  paper  manuals,  the  Navy  would  be  wise  to  devise  a  plan  that 
“stacks  the  deck’’  to  achieve  the  maximum  impact  of  these  benefits  by  considering 
how  IETMs  relate  to  these  other  initiatives. 

Intuitively,  we  believe  that  there  should  be  a  blend  of  investment  in  electronic 
classrooms  and  in  distributed  learning  technology.  The  effective  combination  of 
these  investments  should  reduce  staff  downtime,  enable  independent  learning,  and 
greatly  reduce  training  pipelines.  The  IETM  is  one  technology  common  to  both 
venues.  Integrating  Web-based  training  with  IETMs  serving  as  Electronic  Per¬ 
formance  Support  Systems  (EPSS)  should  support  the  learning  continuum  and 
replace  some  formal  training. 

The  Navy  might  wish  to  determine  the  optimum  mix  of  classroom  modernization 
and  distance  learning  investment.  One  of  the  objectives  of  the  Navy-wide  Distrib¬ 
uted  Learning  Planning  Strategy  is  to  reduce  the  Navy’ s  reliance  on  classroom 
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training.  In  addition,  there  clearly  is  potential  for  training  and  operational  benefit 
to  the  Fleets  by  effectively  employing  IETMs,  Web-based  training,  and  perform¬ 
ance  support  systems.  The  Navy  may  want  to  consider  conducting  low-level  pilots 
to  understand  the  synergistic  effects  of  these  innovations. 

Finally,  when  the  potential  and  implications  of  these  technologies  are  understood, 
the  Navy  may  develop  a  plan  for  cost-effectively  deploying  the  appropriate  sys¬ 
tems  and  establish  realistic  goals  for  cost  savings  and  benefit  returns.  The  Navy 
clearly  could  benefit  from  developing  a  model  that  can  assist  in  the  technology 
assessment  and  selection  process. 

The  objective  of  such  a  model  may  not  be  to  determine  a  categorical  “winner” 
among  discrete  alternatives.  Instead,  it  may  identify  and  assess  the  differences  in 
costs  and  benefits  among  the  alternatives,  with  the  aim  of  characterizing  the 
tradespace  available  to  decision  makers.  Although  this  process  may  not  determine 
an  overall  “best”  alternative,  it  will  suggest  a  way  to  find  the  best  fit  given  deci¬ 
sion  makers’  priorities  and  constraints. 

Because  many  of  the  anticipated  benefits  of  employing  IETMs  in  institutional 
schooling  and  Fleet  environments  cannot  be  quantified  readily  in  monetary  terms, 
we  suggest  that  analysts  not  attempt  to  compute  net  cost-benefit  or  benefit/cost 
ratios.  The  best  way  to  analyze  the  purely  economic  costs  and  benefits  is  by  com¬ 
puting  the  savings/investment  ratio  (SIR).  That  formula  is  as  follows: 

SIR  =  Baseline  Cost-(Cost  of  Alternative  +  Investment) 

Investment 

The  Navy  might  wish  to  use  something  akin  to  the  methodology  in  Figure  5-1  to 
rank  alternative  approaches.  This  methodology  evaluates  combined  “important” 
factors — quantifiable  and  nonquantifiable — as  determined  by  decision  makers. 

Figure  5-1.  Cost-Effectiveness  Model 


Beginning 

State 


A.  Operating  cost 

B.  Classroom  time  reqd 

C.  Throughput 

D.  Student/teacher  ratio 

E.  Student  scores 


Investment: 

-  Design 
Conversion 
Hardware 
Software 
Implementation 


Weight  criteria  (including  investment  cost) 
After/before  ratio  x  weight  =  criteria  score 
E  scores  =  benefit 
Ratio  benefit-to-investment  yields 
comparative  measure 


End 

State 


A.  Operating  cost 

B.  Classroom  time  reqd 

C.  Throughput 

D.  Student/teacher  ratio 

E.  Student  scores 
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Unlike  a  dollar-retum-for-dollar-spent  ROI  model,  this  type  of  framework  enables 
planners  to  establish  the  relative  benefits  of  proposed  technology  insertions.  This 
methodology  simply  is  a  generalization  of  conventional  cost-benefit  analyses  that 
combines  the  contributions  of  multiple  criteria  into  a  single,  unitless  measure  and 
allows  the  analyst  to  rank  options  according  to  the  analyst’s  preferences.  The  cri¬ 
teria— -which  are  notional  here— could  be  determined  through  research,  or  they 
may  be  whatever  “important”  factors  the  user  chooses.  Our  experience  suggests 
that  at  least  one  measure  each  for  cost,  effectiveness,  and  efficiency  would  be  ap¬ 
propriate  for  assessing  the  impact  of  training  technology  investments. 

This  framework  requires  the  user  to  quantify  the  difference  in  these  important 
factors,  from  before  a  technology  insertion  to  after  the  insertion.  It  results  in  a 
benefit  gained  (or  lost)  per  unit  of  cost  for  each  option,  enabling  planners  to  com¬ 
pare  dissimilar  options.  For  a  proposed  technology  insertion,  this  model  assumes 
that  insertion-specific  investment  costs  can  be  estimated  and  that  all  evaluation 
criteria  can  be  quantified,  though  not  necessarily  in  economic  terms.  This  analysis 
may  even  be  on  a  subjective  scale.  Clearly,  an  increase  may  be  desirable  for  some 
criteria  and  undesirable  for  others  (e.g.,  student  throughput  versus  classroom  time 
required).  These  relationships  can  be  accommodated  in  the  criteria  weighting.  If 
desired,  that  same  weighting  system  can  be  used  to  express  all  criteria  in  terms  of 
their  economic  impact  for  a  pure  cost-benefit  assessment. 
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Appendix 

Abbreviations 


AEC 

Advanced  Electronic  Classroom 

ADLI 

Advanced  Distributed  Learning  Initiative 

AIMSS 

Advanced  Integrated  Maintenance  Support  System 

AMIDD 

Aviation  Maintenance  Integrated  Diagnostics 

Demonstration 

asch 

American  Standard  Code  for  Information  Interchange 

ARCI 

Acoustic  Research  COTS  Integration 

ASN(RDA) 

Assistant  Secretary  of  the  Navy  for  Research, 

Development  and  Acquisition 

CALS 

Continuous  Acquisition  and  Life  Cycle  Support 

CLS 

Contract  Logistics  Support 

CNET 

Chief  of  Naval  Education  and  Training 

CONOPS 

concept  of  operations 

COTS 

commercial-off-the-shelf 

EPSS 

Electronic  Performance  Support  System 

ET 

electronics  technician 

ETM 

Electronic  Technical  Manual 

FC 

fire  control  specialist 

GS 

gas  turbine  maintenance  specialist 

HM&E 

hull,  mechanical,  and  electrical 

ICW 

interactive  courseware 

IDP 

Integrated  Development  Program 

EETMPP 

IETM  Process  Plan 

IETM 

Interactive  Electronic  Technical  Manual 

IMAT 

Interactive  Multi-sensor  Analysis  Trainer 

IMI 

interactive  multimedia  instruction 

IT 

information  technology 

NATEC 

Naval  Air  Technical  Data  and  Engineering  Service  Command 
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LRC 

NAVAffi. 

NAVSEA 

NAWC-TSD 

NEC 

NSWC-PHD 

NTC 

O&S 

OJT 

OPNAV 

PDF 

PEDD 

PM 

QA 

ROI 

S/CC/A 

SecNav 

SCORM 

SGML 

SIR 

SPAWAR 

SSC 

SYSCOM 

T/HG 

TAD 

TDMIS 

TFDLAT 

TM 

TMDR 

VTT 

VTUAV 

XML 


Learning  Resource  Center 

Naval  Air  Systems  Command 

Naval  Sea  Systems  Command 

Naval  Air  Warfare  Center-Training  Systems  Division 

Naval  Enlisted  Classification 

Naval  Surface  Warfare  Center-Port  Hueneme  Division 
Naval  Training  Center 
operations  and  support 
on-the-job  training 

Office  of  the  Chief  of  Naval  Operations 
Portable  Document  Format 
Portable  Electronic  Display  Device 
Program  Manager 
quality  assurance 
return  on  investment 
Sonar,  Combat  Control,  Architecture 
Secretary  of  the  Navy 

Shareable  Courseware  Object  Reference  Model 

Standard  Generalized  Markup  Language 

savings/investment  ratio 

Space  and  Naval  Warfare  Systems  Command 

Service  School  Command 

system  command 

Training/IETM  Interface  Guide 

temporary  assigned  duty 

Technical  Data  Management  Information  System 
Total  Force  Advanced  Distance  Learning  Action  Team 
technical  manual 

technical  manual  deficiency  report 
Video  Teletraining 

Vertical  Takeoff  and  Landing  Unmanned  Aerial  Vehicle 
Extensible  Markup  Language 
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